The early bird catches the term: combining twitter and news data for event detection and situational awareness

Problem definition

Our definition of a real-world event within the context of Twitter is taken from [8], with the exception that we have added a concept of event location. We are interested in only those events that attract discussion on Twitter, since all others would be invisible to our methods.

From the above definition, when given a time-ordered stream of Twitter messages M, the event detection problem is therefore one of identifying the events e ₁,…, e _n that are present in this stream and their associated time periods T _e and messages M _e . It is also valuable to identify the primary location or locations L _Mi that messages have originated from, and if possible the event location L _e . The situational awareness problem is one of taking the time period T _e and messages M _e and producing an understandable summary of the event and its context.

Overview

Our approach to the event detection problem incorporates location by detecting deviations from baseline levels of tweet activity in specific geographical areas. This allows us to track the location of messages relating to events, and in some cases determine the event location itself.

Initially the system was designed with disease outbreak detection as the primary use case; this led to a system design focused around keywords and aliases for their keywords, since a limited range of illness symptoms characterises most common diseases and the vocabulary used to describe these symptoms is also relatively limited. After several iterations of this approach we noted that it could be viable as a general event detection and situational awareness method, so we added another type of event to determine the feasibility of the general approach. We chose events causing strong emotions as a contrasting and less specific event type, with the intention of picking up a variety of localisable events such as important football matches and rock concerts.

For each type of event we define a list of keywords, each describing a particular sub-type of that event. For example when looking at illness each keyword relates to a particular symptom, such as ‘coughing’ or ‘vomiting’. When looking at events that cause emotional reactions each keyword relates to a particular emotion. Each of these sub-type keywords is then expanded with a list of aliases, synonyms and related terms to form a keyword group. For more details on how we identified the relevant keywords and synonyms see the sections below.

We track the number of tweets mentioning each keyword (consolidating all that lie in the same keyword group), in each of the geographical areas and use bio-surveillance algorithms to detect spikes in activity. Each spike is treated as a potential event, and we use various criteria to single out those with a high probability of being actual events as defined above, i.e. those that are caused by discussion of real-world occurrences on Twitter.

Our situational awareness approach is based on identifying terms from the event tweets which characterise the events and using them to retrieve relevant news articles and identify the most informative tweets. The news search uses metrics based on cosine similarity to ensure that searches return related groups of articles.

Architecture

The general approach can be described by the architecture in Fig. 1. Every new event type requires a list of keywords and their associated aliases. Optionally a specific data pre-processing step can be included for the event type. For example in the health symptom case we employ a machine learning classifier to remove noise (those tweets not actually concerning health). These are the only two aspects of the design that need to be altered to provide event detection and situational awareness to a new problem domain.

Event types

We now go into a more detailed explanation of our event types and how we formulated the keywords and associated aliases. Each keyword group consists of a primary keyword which is used to identify the group, e.g. vomit, and a number of aliases that expand the group, e.g. throwing up, being sick, etc. see Tables 2 and 3 for the full list of keyword groups.

Data collection

Location assignment

Our methodology relies on the collection of baseline levels of tweet activity in an area, so that alarms can be triggered when this activity increases. We therefore amalgamated the fine-grained location information from the geo-coded tweets by assigning them to broader geographical areas. We used a data driven approach to generate the geographical areas rather than using administrative areas such as towns or counties. This technique allowed us to select only those areas with a minimum level of tweet activity, and also did not require any additional map data. It would therefore be be reusable for any region or country with a sufficient level of Twitter usage.

We began by viewing a sample of the collected tweets as geo-spatial points. Viewed on a map these clearly clustered in the densely populated areas of England and Wales. We therefore decided to use a clustering algorithm on these points in order to separate out areas for study. We employed the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm [23] for clustering, as this does not require a priori knowledge of the number of clusters in the data. The features provided to DBSCAN were the latitudes and longitudes of the tweets.

The clusters produced by the algorithm matched the most populated areas, corresponding to the largest cities in the UK as shown in Fig. 2. They also separated most cities into distinct clusters (a notable exception being the conglomeration of Liverpool and Manchester). In total 39 clusters were created for England and Wales and each was given an ID and a label. We then created a convex hull around each cluster, providing a polygon that can be used to check whether a point is in the cluster or outside it. Points outside all of the clusters were assigned to a special ‘noise’ cluster, and not included in the analysis. Overall 80 % of tweets were assigned to specific clusters and the remainder to noise, giving us good coverage of geo-tagged tweets using our cluster areas.

Tweet processing

As tweets are received by our system they are processed and assigned to the symptom and emotion state classes if they contain one of the relevant keywords. They are assigned a location by checking whether they fall into one of our cluster areas.

For the illness symptoms we introduce a noise removal stage at this point. It is particularly relevant for this class of events because there are many fewer tweets relating to illness than showing emotion states. This means that the signal is more easily blocked out by random noise. To remove noise we construct a machine learning classifier with the aim of removing tweets containing alternative word usages or general illness discussion rather than reporting of illness events. The classifier therefore classifies tweets into those that are self-reports of illness and those that are not. The classifier we use is a linear SVM trained on a semi-supervised cascading training set, created on the principles described in Sadilek et al. [24]. Our classifier uses the LibSVM [25] library, and was initially trained on 4600 manually classified tweets. It achieves a classification accuracy of 96.1 % on a held out test set of 920 manually classified tweets.

The number of tweets assigned to each class in each area are then saved on a daily basis. These counts are first normalised to take account of Twitter’s daily effect pattern, which shows more tweeting on weekends than weekdays. Event detection is run daily since we are attempting to pick up temporally coarse-grained events. Disease outbreaks take weeks to develop, and events that shift public sentiment or emotion will generally take hours or days to unfold.

Detecting events

Our event detection methodology leverages considerable existing syndromic surveillance research by using an algorithm designed and developed by the Centers for Disease Control and Prevention (CDC), the Early Aberration Reporting System (EARS) [26].

We employ the C2 and C3 variants of EARS. These algorithms operate on a time series of count data, which in our case is a count of daily symptomatic tweet activity. The C2 algorithm uses a sliding seven day baseline, and signals an alarm for a time t when the difference between the actual count at t and the moving average at t exceeds 3 standard deviations. The C3 algorithm is based on C2, and in effect triggers when there have been multiple C2 alarms over the previous 3 days.

These C2 and C3 candidate alarms are then grouped together so that alarms for the same keyword group and area on consecutive days are treated as a single alarm. An alarm is therefore made up of one or more days, each with an observed count of tweets. An alarm ends when the C2 and C3 algorithms no longer signal an outbreak occurring.

The μ _max is the primary statistic which we use to determine which events are real and which have just been generated by random noise.

Another statistic which we employ in order to filter out noise is the tweet-user ratio. This is the ratio of tweets in an event to that of distinct users involved in an event. A high value of this statistic would imply that some users have tweeted a large number of times across a short time period, which is an indication that they may be spammers and that the alarm is spurious.

In summary, we use the output from EARS to produce alarms. We filter the alarms to a set of high likelihood events by using the μ _max and tweet-user ratio parameters. From this point we refer to those alarms that are high-likelihood as events, according to our earlier event definition. The alarms have an associated stream of Twitter messages and a location given by the node which they occur in. The following situational awareness results show that the Twitter messages in these alarms discuss real-world occurrences, therefore fulfilling all of our definition.

Situational awareness

Once an event has been identified our next objective is to automatically provide additional context for it, which may provide an explanation of the underlying cause. A human interpreter could achieve this by reading all of the tweets and synthesizing them into a textual explanation, which might be some text such as “People reacting to the death of Robin Williams”. We do this in two main ways: by providing the most representative tweets from those that triggered the alarm, and by linking to relevant news articles. The steps involved in the Terms, News and Tweets (TNT) Event Summarisation process are detailed in Algorithm 1. The steps and terminology are then explained in more detail.

① The first step is to retrieve the relevant tweets from the processed tweet and alarm databases. Tweets are fetched for both the alarm gist and from a historical baseline. ③ We discard those events with fewer than 30 tweets as we found that they did not contain sufficient data to produce good summarisation results.

⑤ The next task is to find unigrams and bigrams that are more prevalent in the gist than in the baseline. These are likely to come from tweets discussing the event and will thus be characteristic of the event. We first extract the most common unigrams and bigrams from both sets of tweets, after removal of stopwords. Our list of stopwords includes a standard list, plus the 200 most frequent words from our tweet database. We select all non-stopwords that appear in at least 5 % of the tweets.

⑦ We then do a Fisher’s Exact Test to determine which of the common unigrams and bigrams in the gist appear significantly more frequently (α<0.05) here than in the baseline set. Our candidate terms are the top two most significant unigrams and bigrams. We select the top two as this was found to give the best results on our test examples. To this set we append the primary keyword that triggered the alarm.

⑨ For this research Google was used as the news database. Using the candidate terms we perform a search on Google for documents published in the United Kingdom during the time period of the alarm. Due to Google’s Terms of Service this step was performed manually. A fully automated system would replace this step with a search of a news database, which could be created by pulling down news articles from RSS feeds of major content providers.

The PCSS rewards articles which are similar and penalises any variance across those article similarities. This reduces the effect of some articles being strongly related in the document set and others being highly unrelated. Any term which retrieves a set of documents with a score below a threshold value is not considered further.

It is possible for a search term to hit on a coherent set of documents purely by chance, perhaps by finding news articles related to another event in a different part of the world. In order to guard against this we institute another check to ensure that the set of documents returned from a search term is sufficiently closely related to the set returned from at least one other search term.

⑫ In order to perform this check we compare the titles of the articles returned from the two different searches using a similar process to our earlier document comparison. We found it more effective to compare titles than whole documents, since sets of documents with similar topics can contain similar language even for fairly unrelated search terms. For example the terms “ebola” and “flu” will both return health-related documents containing similar language, but we would not wish to say that these search terms are related. To convert the titles to TF/IDF vectors we remove stopwords but do not apply stemming. Since the titles are so short we include all unigrams, bigrams and trigrams in the vector representation. We then compute a PCSS between the two document sets, pairing each document in the first set with each in the second and vice versa. ⑬ A search term must be related to at least one other term for it to be used going forward.

⑭ Once TNT has identified good search terms we then return the news articles fetched using those terms. ⑯ In order to rank the top news articles for a search we take the mean TF/IDF vector of the articles. and then rank the articles by cosine similarity to this mean vector. We return the top ranked articles from each search term.

1) is always available and is labelled the Gist Top Tweets (GTT). If the TNT algorithm has found terms that are significantly different in frequency from the baseline then set 2) is available for use and if terms from that set have good news matches then set 3) can be used. The Summary Top Tweets (STT) are from set 3) if it exists and fallback to set 2) if the good news match terms are not available. If no terms were found to be significantly different from the baseline then only the GTT is available.

In order to choose the top tweets we rank them by their cosine similarity to the mean TF/IDF vector of all tweets in the set, an approach similar to that of [4]. This attempts to finds tweets which capture and summarise the aggregate information of all of those in the set. The top five tweets ranked by this measure are returned.

Candidate event selection

Over the course of the study the bio-surveillance algorithms generated 820 disease-related alarms and 2021 emotion-related alarms. A brief survey of these revealed that many were false alarms generated by random fluctuations in the noisy social media data. In order to separate out alarms that could be labelled as events with high confidence we conducted the following analysis.

Firstly we compiled an initial set of 13 focus example alarms. These were taken from events that the authors knew had happened in the evaluation time period and from those alarms in our dataset with low and high values of μ _max .

The most important threshold parameter in the context of the event detection is the μ _max figure which measures the deviation of the alarm counts from the median level. Examining the distribution of the number of alarms for each value of μ _max revealed that it started to tail off sharply at μ _max ≥5. The distribution of alarms for each value of μ _max is shown in Fig. 3. We therefore took this as a value to segment additional test examples, drawing ten more at random with a μ _max less than 5 and ten with a μ _max greater than or equal to 5. The resulting evaluation set of 33 candidate events is shown in Table 4. The event ID used to refer to the events is composed of the first two letters of the event keyword followed by a 1–2 letter area code. The final part of the ID is the day and month of the event start date.

ID	Event	μ max	Keyword	Node	ID	Event	μ max	Keyword	Node
SAL-11-08	YES	20	Sadness	London	HFB-10-04	YES	5	Hayfever	Birmingham
HFM-01-06	YES	19	Hayfever	Manchester	VOL-20-04	YES	5	Vomit	London
SAL-07-04	YES	14	Sadness	London	SAC-05-05	YES	5	Sadness	Cardiff
FEL-18-07	YES	13	Fear	London	HFL-04-07	NO	5	Hayfever	London
ASL-02-04	YES	12	Asthma	London	FLB-23-09	NOa	5	Flu	Birmingham
FLP-06-10	YES	11	Flu	Portsmouth	VPBR-10-05	YES	4	VeryPos	Bristol
HAM-02-04	YES	9	Happy	Manchester	FRL-30-05	YES	4	Fever	London
HAM-18-04	YES	9	Happy	Manchester	FLM-19-09	YES	4	Flu	Manchester
SAL-08-07	YES	8	Sadness	London	VOL-22-02	NO	3	Vomit	London
HALE-01-08	YES	8	Happy	Leeds	HFB-29-04	NO	3	Hayfever	Birmingham
HFL-14-05	YES	7	Hayfever	Leeds	JONO-23-02	YES	2	Joy	Norwich
SUN-29-08	YES	7	Surprise	Newcastle	HEM-06-03	NO	2	Headache	Manchester
ITL-08-06	YES	6	Itch	London	SUC-23-05	NO	2	Surprise	Cardiff
SAB-09-06	YES	6	Sadness	Birmingham	SUL-16-08	NO	1	Surprise	London
HABE-01-03	YES	5	Happy	Bridgend	FEBR-17-04	NO	0	Fear	Bristol
SAL-21-03	YES	5	Sadness	London	STL-26-08	NO	0	Sore Throat	London
HFC-09-04	YES	5	Hayfever	Cardiff

Shows whether events were externally verifiable and their μ _max value

^aNote: this event not confirmed by the GP in hours report of that week. However, the following week showed an increase and it is possible that social media detected increased Influenza activity before this was confirmed by GP visits

Event detection evaluation

It is difficult to provide a completely automated evaluation procedure for detecting previously unknown events. Diaz et al. used the time to detection on a known outbreak as their evaluation criterion [15]. In our case we do not know a priori that these are genuine outbreaks or events. Hence we need to make an assessment of the alarms produced to see what they refer to and if there is a way of externally verifying that they are genuine events. For all 33 of the selected alarms the authors read the tweets and determined whether they described a real world event. The coders found 26 YES answers, 5 NO answers and 2 DISAGREED answers, producing a 94 % agreement. Where an event was present they wrote a short summary.

For external verification of events two different methods were used, depending on whether the event was symptom-related or emotion-based. For symptom related events the activity spike was checked against official sources for the same time period. The General Practitioner (GP) in hours bulletin for England and Wales [27] was used and an event was deemed verified if the symptom exhibited an increasing trend for that period. This detail is noted in the summary document produced by Public Health England for that reporting period. Emotion-based events were verified by checking if there were any articles (via Web search) that could corroborate the cause of the event (as given by the summary).

We manually investigated all examples from the initial focus set and found initial parameters for the score functions in our algorithms that worked reasonably well. These provided possible ranges of values which were evaluated more systematically over the entire alarm set. For event detection we evaluated which alarms were flagged as events by the system for each parameter value against whether those events were externally verifiable. The final evaluation for all algorithms contains all 33 of the alarms in both sets, not just the twenty expanded ‘test’ examples.

To determine if an alarm is an event that we should be concerned about we consider two properties of the alarm. The first is the tweet-user ratio. From exploratory testing we found a value of 1.5 separated our spam and genuine alarms very well, leaving only a small number of alarms with large tweet sets and some spam. The spam detection problem should be straightforward and will be addressed more completely in future work.

The second figure which gives the strength of the activity above the usual baseline is the μ _max figure. This is the essence of the modified EARS algorithm and the value of this figure should generally separate events from non-events.

The precision, recall and F1 values for all the tested values of μ _max are displayed in Fig. 4. All figures were calculated with reference to the set of 33 example events discussed above. The maximum F1 value, 0.9362, is observed at μ _max ≥4, so this is a well balanced threshold and the recommended parameter. Those seeking higher confidence events (willing to accept that some events may be missed) could use a value of 6 for this parameter which yields a precision of 1. The maximum observed recall value is at the minimum parameter value and is not very informative. Essentially it says that everything is an event and hence does not produce any false negatives.

In summary the event detection mechanism based on the EARS C2 and C3 algorithms with the addition of the μ _max and tweet-user ratio was found to perform well at detecting events that could be externally verified as genuine. The recommended μ _max parameter (4) produced a good balance of precision and recall in our sample set. It must be noted however that we cannot gain a true picture of the overall recall of the system, since we have no way of analysing the number of genuine events that were not picked up.

Situational awareness evaluation

Both situational awareness components were evaluated. Firstly the news linkage was tested to see whether relevant news was retrieved for the sample events. As part of this analysis we compared our method of extracting informative search terms (the TNT algorithm) with a comparable automated technique. Secondly the tweet ranking was validated to determine whether highly ranked tweets effectively summarised the events.

Comparative news linkage evaluation

The news linkage component works by selecting good search terms for articles based on the TNT algorithm. Within this there is a term extraction step to generate search terms, and then a filtering step using PCSS to remove terms which retrieve unrelated sets of articles. We iterate over different threshold values for the PCSS score to find the optimum, using an F0.5 measure as the evaluation criterion. F0.5 was selected because precision was judged to be more important than recall in this setting. As a further evaluation we compare the results of replacing our term extraction algorithm with Latent Dirichlet Allocation (LDA). LDA is a popular topic modelling technique that extracts sets of terms characterising each topic in a group of documents. The success and error types used to compute the F0.5 measure are:

• True positive: relevant news returned for newsworthy event

• False positive: news returned for an event with no genuine news

• True negative: no news returned for an event with no genuine news

• False negative: no news returned for newsworthy event

The evaluation is presented in Figs. 5 and 6 as well as the different levels of article PCSS that were iterated over to find the maximum F0.5 value in a step-wise procedure. It is clear from these images that the TNT algorithm has a higher F0.5 at all tested values of the article PCSS, due to its higher recall. The outcome of the parameter selection process was that a PCSS threshold of −0.08 produced the best results. Using this value the F0.5 was 0.79, showing that our system was successful in retrieving relevant news for the sample events.

Selecting top ranked relevant news articles is one part of our situational awareness contribution. The second is the selection of tweets that provide a representative summary of an event.

Notable examples discussion

It is difficult for a term-based solution to find any common thread here. Finding the cause of this event would require contextual knowledge of football matches, team names and commonly employed aliases. The news linkage algorithm did initially find a news story for the term “joy” on this date. The British Prime Minister “let out a little cry of joy” over David Bowie Scottish independence comments (Telegraph, Feb 24, 2014). The articles returned all concerned this story and were found to be closely related, but were dropped from the news linkage because they did not match those returned from the other search terms. This highlights the benefits of searching with multiple terms and ensuring that the results are related.

Here selecting the top tweets from the filtered event set captures tweets representative of the event as opposed to the baseline illness activity. The news linkage for this example worked well, with all five of the top selected articles being representative of the event. The top article, “Air pollution reaches high levels in parts of England - BBC”, gives the cause of the event in the first few lines: “People with health problems have been warned to take particular care because of the pollution - a mix of local emissions and dust from the Sahara.”

While the top ranked tweets are similar the event tweet filtering does remove baseline tweets referring to general illness. No good news searches were found in this case. This event may be valid in the context of social media but it is not newsworthy.