Temporal Web Image Retrieval Gal Dias a , Jos G. Moreno a , Adam - - PowerPoint PPT Presentation

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Retrieval

Gaël Diasa, José G. Morenoa, Adam Jatowtb, Ricardo Camposc,( Paul Martina, Frédéric Juriea, Youssef Chahira)

(a) HULTECH/IMAGE/GREYC - University of Caen Basse-Normandie, France (b) TANAKA Lab - University of Kyoto, Japan (c) LIAAD-INESC TEC - Polytechnic Institute of Tomar, Portugal

SPIRE/LAWEB 2012 Cartagena de Indias, Colombia October 25th

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Outline

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Outline

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Information Retrieval in Time

◮ Classical IR: Given a query, Retrieve and Rank the most

relevant documents.

◮ New needs in IR: Given a query, Retrieve, Rank, Filter and

Organize the most relevant documents based on different dimensions.

◮ Different Dimensions in Web Search: Multi-faceted,

Personnalized, Collaborative, Opinion, Freshness, Diversity, Spatial, Temporal, etc.

◮ Temporal Web Search: Retrieve, Organize and Filter the

most relevant documents in terms of Temporal intents.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Information Retrieval in Time

◮ Classical IR: Given a query, Retrieve and Rank the most

relevant documents.

◮ New needs in IR: Given a query, Retrieve, Rank, Filter and

Organize the most relevant documents based on different dimensions.

◮ Different Dimensions in Web Search: Multi-faceted,

Personnalized, Collaborative, Opinion, Freshness, Diversity, Spatial, Temporal, etc.

◮ Temporal Web Search: Retrieve, Organize and Filter the

most relevant documents in terms of Temporal intents.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Information Retrieval in Time

◮ Classical IR: Given a query, Retrieve and Rank the most

relevant documents.

◮ New needs in IR: Given a query, Retrieve, Rank, Filter and

Organize the most relevant documents based on different dimensions.

◮ Different Dimensions in Web Search: Multi-faceted,

Personnalized, Collaborative, Opinion, Freshness, Diversity, Spatial, Temporal, etc.

◮ Temporal Web Search: Retrieve, Organize and Filter the

most relevant documents in terms of Temporal intents.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Information Retrieval in Time

◮ Classical IR: Given a query, Retrieve and Rank the most

relevant documents.

◮ New needs in IR: Given a query, Retrieve, Rank, Filter and

Organize the most relevant documents based on different dimensions.

◮ Different Dimensions in Web Search: Multi-faceted,

Personnalized, Collaborative, Opinion, Freshness, Diversity, Spatial, Temporal, etc.

◮ Temporal Web Search: Retrieve, Organize and Filter the

most relevant documents in terms of Temporal intents.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Information Retrieval

◮ Temporal Information Retrieval (TIR) aims to present

information along its temporal dimension.

◮ One of the most important motivations for Textual TIR is

the creation of timelines of major events.

◮ Some motivations for Visual TIR are to understand the

evolution of a city or a place, or observe changes in person’s outlook.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Information Retrieval

◮ Temporal Information Retrieval (TIR) aims to present

information along its temporal dimension.

◮ One of the most important motivations for Textual TIR is

the creation of timelines of major events.

◮ Some motivations for Visual TIR are to understand the

evolution of a city or a place, or observe changes in person’s outlook.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Information Retrieval

◮ Temporal Information Retrieval (TIR) aims to present

information along its temporal dimension.

◮ One of the most important motivations for Textual TIR is

the creation of timelines of major events.

◮ Some motivations for Visual TIR are to understand the

evolution of a city or a place, or observe changes in person’s outlook.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

What We (would like to) Obtain (Cartagena de Indias)

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

While Google Gives Us This (Cartagena de Indias)

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (I)

◮ Many studies have been appearing in the past 5 years

based on Textual Data.

◮ Foundations of Temporal IR: (Baeza-Yates, 2005). ◮ Query Temporal Disambiguation: (Jones and Diaz, 2007),

(Metzler et al., 2009).

◮ Temporal Clustering: (Alonso et al., 2009), (Campos et al.,

2012).

◮ Temporal Ranking: (Kanhabua et al., 2011), (Chang et al.,

2012).

◮ Temporal Language Models: (Berberich et al., 2010). ◮ Future IR: (Jatowt and Yeung, 2011), (Dias et al., 2011). Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (II)

◮ To our knowledge, only two studies based on Visual Data

have been proposed so far.

◮ Temporal classification of web images (Palermo, 2012). ◮ Temporal ephemeral clustering and classification of web

images (SPIRE 2012).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (II)

◮ To our knowledge, only two studies based on Visual Data

have been proposed so far.

◮ Temporal classification of web images (Palermo, 2012). ◮ Temporal ephemeral clustering and classification of web

images (SPIRE 2012).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Multimedia Temporal Information Retrieval (II)

◮ To our knowledge, only two studies based on Visual Data

have been proposed so far.

◮ Temporal classification of web images (Palermo, 2012). ◮ Temporal ephemeral clustering and classification of web

images (SPIRE 2012).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Outline

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Ephemeral Clustering (I)

◮ Ephemeral clustering is also known as Search Results

Clustering (SRC).

◮ SRC algorithms deal with small data sets such as web

search results (e.g. web snippets, web images).

◮ SRC algorithms focus on the dynamics and volatility of the

web.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Ephemeral Clustering (I)

◮ Ephemeral clustering is also known as Search Results

Clustering (SRC).

◮ SRC algorithms deal with small data sets such as web

search results (e.g. web snippets, web images).

◮ SRC algorithms focus on the dynamics and volatility of the

web.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Ephemeral Clustering (I)

◮ Ephemeral clustering is also known as Search Results

Clustering (SRC).

◮ SRC algorithms deal with small data sets such as web

search results (e.g. web snippets, web images).

◮ SRC algorithms focus on the dynamics and volatility of the

web.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Ephemeral Clustering (II)

◮ Each web image, together with its text web snippet,

contains an endogenous:

◮ thematic dimension(s), ◮ temporal dimension(s).

◮ SRC algorithms should successfully build topically and

temporally related clusters of web images.

◮ A cluster should only contain web images, which focus on

the same topic in the same period of time.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Ephemeral Clustering (II)

◮ Each web image, together with its text web snippet,

contains an endogenous:

◮ thematic dimension(s), ◮ temporal dimension(s).

◮ SRC algorithms should successfully build topically and

temporally related clusters of web images.

◮ A cluster should only contain web images, which focus on

the same topic in the same period of time.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Ephemeral Clustering (II)

◮ Each web image, together with its text web snippet,

contains an endogenous:

◮ thematic dimension(s), ◮ temporal dimension(s).

◮ SRC algorithms should successfully build topically and

temporally related clusters of web images.

◮ A cluster should only contain web images, which focus on

the same topic in the same period of time.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Topical Ephemeral Clustering

◮ Topical Ephemeral Clustering is also commonly named

Multi-Faceted Search.

◮ Many works exist over web snippets (Carpineto et al,

2009), (Dias et al, 2011), (Scaiella et al, 2012).

◮ Works over web images have been following a three-steps

procedure (Din et al, 2008), (Moreno et al, 2011).

◮ For a given query, relevant facets are embodied by the

cluster labels obtained by SRC algorithms over (text) web image snippets,

◮ Topical Clustering is obtained by repetitive facet query

expansion,

◮ Images with similar (topical) visual contents are maintained

within clusters.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Topical Ephemeral Clustering

◮ Topical Ephemeral Clustering is also commonly named

Multi-Faceted Search.

◮ Many works exist over web snippets (Carpineto et al,

2009), (Dias et al, 2011), (Scaiella et al, 2012).

◮ Works over web images have been following a three-steps

procedure (Din et al, 2008), (Moreno et al, 2011).

◮ For a given query, relevant facets are embodied by the

cluster labels obtained by SRC algorithms over (text) web image snippets,

◮ Topical Clustering is obtained by repetitive facet query

expansion,

◮ Images with similar (topical) visual contents are maintained

within clusters.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Topical Ephemeral Clustering

◮ Topical Ephemeral Clustering is also commonly named

Multi-Faceted Search.

◮ Many works exist over web snippets (Carpineto et al,

2009), (Dias et al, 2011), (Scaiella et al, 2012).

◮ Works over web images have been following a three-steps

procedure (Din et al, 2008), (Moreno et al, 2011).

◮ For a given query, relevant facets are embodied by the

cluster labels obtained by SRC algorithms over (text) web image snippets,

◮ Topical Clustering is obtained by repetitive facet query

expansion,

◮ Images with similar (topical) visual contents are maintained

within clusters.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Topical Ephemeral Clustering

◮ Topical Ephemeral Clustering is also commonly named

Multi-Faceted Search.

◮ Many works exist over web snippets (Carpineto et al,

2009), (Dias et al, 2011), (Scaiella et al, 2012).

◮ Works over web images have been following a three-steps

procedure (Din et al, 2008), (Moreno et al, 2011).

◮ For a given query, relevant facets are embodied by the

cluster labels obtained by SRC algorithms over (text) web image snippets,

◮ Topical Clustering is obtained by repetitive facet query

expansion,

◮ Images with similar (topical) visual contents are maintained

within clusters.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Topical Ephemeral Clustering

◮ Topical Ephemeral Clustering is also commonly named

Multi-Faceted Search.

◮ Many works exist over web snippets (Carpineto et al,

2009), (Dias et al, 2011), (Scaiella et al, 2012).

◮ Works over web images have been following a three-steps

procedure (Din et al, 2008), (Moreno et al, 2011).

◮ For a given query, relevant facets are embodied by the

cluster labels obtained by SRC algorithms over (text) web image snippets,

◮ Topical Clustering is obtained by repetitive facet query

expansion,

◮ Images with similar (topical) visual contents are maintained

within clusters.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Topical Ephemeral Clustering

◮ Topical Ephemeral Clustering is also commonly named

Multi-Faceted Search.

◮ Many works exist over web snippets (Carpineto et al,

2009), (Dias et al, 2011), (Scaiella et al, 2012).

◮ Works over web images have been following a three-steps

procedure (Din et al, 2008), (Moreno et al, 2011).

◮ For a given query, relevant facets are embodied by the

cluster labels obtained by SRC algorithms over (text) web image snippets,

◮ Topical Clustering is obtained by repetitive facet query

expansion,

◮ Images with similar (topical) visual contents are maintained

within clusters.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Ephemeral Clustering

◮ Temporal Ephemeral Clustering has been proposed by

(Alonso et al., 2009) and (Campos et al., 2012) in the context of web snippet.

◮ Works on Temporal Ephemeral Clustering for web images

do not exist.

◮ We propose a three-steps procedure:

◮ For a given query, relevant year dates are extracted based

• n auto-completion engines,

◮ Temporal Clustering is obtained by repetitive temporal

query expansion,

◮ Images with similar temporal visual intents are maintained

within clusters (Classification Problem).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Ephemeral Clustering

◮ Temporal Ephemeral Clustering has been proposed by

(Alonso et al., 2009) and (Campos et al., 2012) in the context of web snippet.

◮ Works on Temporal Ephemeral Clustering for web images

do not exist.

◮ We propose a three-steps procedure:

◮ For a given query, relevant year dates are extracted based

• n auto-completion engines,

◮ Temporal Clustering is obtained by repetitive temporal

query expansion,

◮ Images with similar temporal visual intents are maintained

within clusters (Classification Problem).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Ephemeral Clustering

◮ Temporal Ephemeral Clustering has been proposed by

(Alonso et al., 2009) and (Campos et al., 2012) in the context of web snippet.

◮ Works on Temporal Ephemeral Clustering for web images

do not exist.

◮ We propose a three-steps procedure:

◮ For a given query, relevant year dates are extracted based

• n auto-completion engines,

◮ Temporal Clustering is obtained by repetitive temporal

query expansion,

◮ Images with similar temporal visual intents are maintained

within clusters (Classification Problem).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Ephemeral Clustering

◮ Temporal Ephemeral Clustering has been proposed by

(Alonso et al., 2009) and (Campos et al., 2012) in the context of web snippet.

◮ Works on Temporal Ephemeral Clustering for web images

do not exist.

◮ We propose a three-steps procedure:

◮ For a given query, relevant year dates are extracted based

• n auto-completion engines,

◮ Temporal Clustering is obtained by repetitive temporal

query expansion,

◮ Images with similar temporal visual intents are maintained

within clusters (Classification Problem).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Ephemeral Clustering

◮ Temporal Ephemeral Clustering has been proposed by

(Alonso et al., 2009) and (Campos et al., 2012) in the context of web snippet.

◮ Works on Temporal Ephemeral Clustering for web images

do not exist.

◮ We propose a three-steps procedure:

◮ For a given query, relevant year dates are extracted based

• n auto-completion engines,

◮ Temporal Clustering is obtained by repetitive temporal

query expansion,

◮ Images with similar temporal visual intents are maintained

within clusters (Classification Problem).

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Temporal Web Image Clustering Framework

Input: TextQuery, TemporalIntentDetectorj; Output: TemporalImageClusterSet; QueryTemporalIntentSet = TemporalIntentDetectorj(TextQuery); for each QueryTemporalIntenti in QueryTemporalIntentSet do ClusterYearName = getYearIntent(QueryTemporalIntenti); ExpandedTemporalQuery = concat(TextQuery, ClusterYearName); TemporalImageClusteri = getImageResults(ExpandedTemporalQuery); TemporalImageClusterNamei = ClusterYearName; TemporalVisualFiltering(TemporalImageClusteri); end for return TemporalImageClusterSet;

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Some Other Results

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Outline

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Principles of Temporal Classification

◮ Given a digital document (text or image), predict the

creation date of the document.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Works on Temporal Classification

◮ Within text books, some approaches have been proposed

based on Language Models (De Jong et al, 2005) and (Kanhabua et al, 2009).

◮ Within images, the only known approach is the one

proposed by (Palermo et al., 2012) based on dating color images with SVM between 1930 and 1980.

◮ There are early works based on manual dating (Coe, 1983),

who analyse the support (e.g. paper type, size) and visual characteristics of objects, places or people in the pictures.

◮ We should also mention a Kodak 2010 patent, which dates

photos based on distinguishing marks that may appear on the back of the photo.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Works on Temporal Classification

◮ Within text books, some approaches have been proposed

based on Language Models (De Jong et al, 2005) and (Kanhabua et al, 2009).

◮ Within images, the only known approach is the one

proposed by (Palermo et al., 2012) based on dating color images with SVM between 1930 and 1980.

◮ There are early works based on manual dating (Coe, 1983),

who analyse the support (e.g. paper type, size) and visual characteristics of objects, places or people in the pictures.

◮ We should also mention a Kodak 2010 patent, which dates

photos based on distinguishing marks that may appear on the back of the photo.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Works on Temporal Classification

◮ Within text books, some approaches have been proposed

based on Language Models (De Jong et al, 2005) and (Kanhabua et al, 2009).

◮ Within images, the only known approach is the one

proposed by (Palermo et al., 2012) based on dating color images with SVM between 1930 and 1980.

◮ There are early works based on manual dating (Coe, 1983),

who analyse the support (e.g. paper type, size) and visual characteristics of objects, places or people in the pictures.

◮ We should also mention a Kodak 2010 patent, which dates

photos based on distinguishing marks that may appear on the back of the photo.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Works on Temporal Classification

◮ Within text books, some approaches have been proposed

based on Language Models (De Jong et al, 2005) and (Kanhabua et al, 2009).

◮ Within images, the only known approach is the one

proposed by (Palermo et al., 2012) based on dating color images with SVM between 1930 and 1980.

◮ There are early works based on manual dating (Coe, 1983),

who analyse the support (e.g. paper type, size) and visual characteristics of objects, places or people in the pictures.

◮ We should also mention a Kodak 2010 patent, which dates

photos based on distinguishing marks that may appear on the back of the photo.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

What Can We Achieve Easily?

◮ A set of 1170 web images based on 5 city names queries

extracted from Flickr.

◮ Five classes based on the evolution of photography:

[1826, 1925), [1925, 1948), [1948, 1968), [1968, 1982), [1982, 2011].

◮ Color and texture features: ScalableColor, FCTH and

CEDD.

◮ 10-fold Cross-Validation for MultiClass SVM with Linear

Kernel (default parameters).

◮ Average F−Measure of 0.509 is achieved.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

What Can We Achieve Easily?

◮ A set of 1170 web images based on 5 city names queries

extracted from Flickr.

◮ Five classes based on the evolution of photography:

[1826, 1925), [1925, 1948), [1948, 1968), [1968, 1982), [1982, 2011].

◮ Color and texture features: ScalableColor, FCTH and

CEDD.

◮ 10-fold Cross-Validation for MultiClass SVM with Linear

Kernel (default parameters).

◮ Average F−Measure of 0.509 is achieved.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

What Can We Achieve Easily?

◮ A set of 1170 web images based on 5 city names queries

extracted from Flickr.

◮ Five classes based on the evolution of photography:

[1826, 1925), [1925, 1948), [1948, 1968), [1968, 1982), [1982, 2011].

◮ Color and texture features: ScalableColor, FCTH and

CEDD.

◮ 10-fold Cross-Validation for MultiClass SVM with Linear

Kernel (default parameters).

◮ Average F−Measure of 0.509 is achieved.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

What Can We Achieve Easily?

◮ A set of 1170 web images based on 5 city names queries

extracted from Flickr.

◮ Five classes based on the evolution of photography:

[1826, 1925), [1925, 1948), [1948, 1968), [1968, 1982), [1982, 2011].

◮ Color and texture features: ScalableColor, FCTH and

CEDD.

◮ 10-fold Cross-Validation for MultiClass SVM with Linear

Kernel (default parameters).

◮ Average F−Measure of 0.509 is achieved.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

What Can We Achieve Easily?

◮ A set of 1170 web images based on 5 city names queries

extracted from Flickr.

◮ Five classes based on the evolution of photography:

[1826, 1925), [1925, 1948), [1948, 1968), [1968, 1982), [1982, 2011].

◮ Color and texture features: ScalableColor, FCTH and

CEDD.

◮ 10-fold Cross-Validation for MultiClass SVM with Linear

Kernel (default parameters).

◮ Average F−Measure of 0.509 is achieved.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Fine-Grained Temporal Web Image Classification (I)

◮ The idea is to find the changes of visual characteristics

based on one single topic following a real-world year distribution.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Fine-Grained Temporal Web Image Classification (II)

◮ A set of 1093 web images about streets extracted from

Flickr (The Commons) to train the model.

◮ A set of 8831 web images to test the model. ◮ 5, 23 and 36 classes from 1820 to 1999 (i.e. 36, 8 and 5

years period).

◮ Color and texture features (LAB, SIFT) with visual words

discovery (for both characteristics) and spatial pyramid transformation (only for SIFT).

◮ One against the Rest SVM with Linear Kernel.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Fine-Grained Temporal Web Image Classification (II)

◮ A set of 1093 web images about streets extracted from

Flickr (The Commons) to train the model.

◮ A set of 8831 web images to test the model. ◮ 5, 23 and 36 classes from 1820 to 1999 (i.e. 36, 8 and 5

years period).

◮ Color and texture features (LAB, SIFT) with visual words

discovery (for both characteristics) and spatial pyramid transformation (only for SIFT).

◮ One against the Rest SVM with Linear Kernel.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Fine-Grained Temporal Web Image Classification (II)

◮ A set of 1093 web images about streets extracted from

Flickr (The Commons) to train the model.

◮ A set of 8831 web images to test the model. ◮ 5, 23 and 36 classes from 1820 to 1999 (i.e. 36, 8 and 5

years period).

◮ Color and texture features (LAB, SIFT) with visual words

discovery (for both characteristics) and spatial pyramid transformation (only for SIFT).

◮ One against the Rest SVM with Linear Kernel.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Fine-Grained Temporal Web Image Classification (II)

◮ A set of 1093 web images about streets extracted from

Flickr (The Commons) to train the model.

◮ A set of 8831 web images to test the model. ◮ 5, 23 and 36 classes from 1820 to 1999 (i.e. 36, 8 and 5

years period).

◮ Color and texture features (LAB, SIFT) with visual words

discovery (for both characteristics) and spatial pyramid transformation (only for SIFT).

◮ One against the Rest SVM with Linear Kernel.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Fine-Grained Temporal Web Image Classification (II)

◮ A set of 1093 web images about streets extracted from

Flickr (The Commons) to train the model.

◮ A set of 8831 web images to test the model. ◮ 5, 23 and 36 classes from 1820 to 1999 (i.e. 36, 8 and 5

years period).

◮ Color and texture features (LAB, SIFT) with visual words

discovery (for both characteristics) and spatial pyramid transformation (only for SIFT).

◮ One against the Rest SVM with Linear Kernel.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Results of the Second Experiment

◮ For 26, 8, 5 years period classes, Accuracy of 0.778, 0.461

and 0.382 are achieved.

◮ “Comparatively”, (Palermo et al., 2012) achieve Accuracy

• f 0.457 for 10 years period classes on color images.

◮ Humans only reach 0.260 Accuracy!

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Results of the Second Experiment

◮ For 26, 8, 5 years period classes, Accuracy of 0.778, 0.461

and 0.382 are achieved.

◮ “Comparatively”, (Palermo et al., 2012) achieve Accuracy

• f 0.457 for 10 years period classes on color images.

◮ Humans only reach 0.260 Accuracy!

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Results of the Second Experiment

◮ For 26, 8, 5 years period classes, Accuracy of 0.778, 0.461

and 0.382 are achieved.

◮ “Comparatively”, (Palermo et al., 2012) achieve Accuracy

• f 0.457 for 10 years period classes on color images.

◮ Humans only reach 0.260 Accuracy!

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Outline

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Conclusions

◮ bla

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Outline

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

Future Works

◮ bla.

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval

Context Temporal Web Image Ephemeral Clustering Temporal Web Image Classification Conclusions Future Works

THANK YOU!

A new hard task to deal with :)

Gaël Dias, José G. Moreno, Adam Jatowt, Ricardo Campos, et al. HULTECH/IMAGE/GREYC, TANAKA Lab, LIAAD-INESC TEC Temporal Web Image Retrieval