Matters per Class Utilizing BERTopic. Find out how to perceive the variations in… | by Mariya Mansurova | Sep, 2023

You’ll find the complete code on GitHub.

Based on the documentation, we usually don’t must preprocess knowledge except there may be a variety of noise, for instance, HTML tags or different markdowns that don’t add which means to the paperwork. It’s a big benefit of BERTopic as a result of, for a lot of NLP strategies, there may be a variety of boilerplate to preprocess your knowledge. In case you are keen on the way it may appear like, see this information for Matter Modelline utilizing LDA.

You should utilize BERTopic with knowledge in a number of languages specifying BERTopic(language= "multilingual"). Nonetheless, from my expertise, the mannequin works a bit higher with texts translated into one language. So, I’ll translate all feedback into English.

For translation, we are going to use deep-translator package deal (you possibly can set up it from PyPI).

Additionally, it may very well be fascinating to see distribution by languages, for that we may use langdetect package deal.

import langdetect
from deep_translator import GoogleTranslator
def get_language(textual content):
attempt:
return langdetect.detect(textual content)
besides KeyboardInterrupt as e:
increase(e)
besides:
return '<-- ERROR -->'
def get_translation(textual content):
attempt:
return GoogleTranslator(supply='auto', goal='en')
.translate(str(textual content))
besides KeyboardInterrupt as e:
increase(e)
besides:
return '<-- ERROR -->'
df['language'] = df.evaluation.map(get_language)
df['reviews_transl'] = df.evaluation.map(get_translation)

In our case, 95+% of feedback are already in English.

To grasp our knowledge higher, let’s have a look at the distribution of critiques’ size. It reveals that there are a variety of extraordinarily quick (and probably not significant feedback) — round 5% of critiques are lower than 20 symbols.

We will have a look at the most typical examples to make sure that there’s not a lot data in such feedback.

df.reviews_transl.map(lambda x: x.decrease().strip()).value_counts().head(10)
critiques
none                          74
<-- error -->                 37
nice lodge                   12
excellent                        8
wonderful worth for cash      7
good worth for cash           7
excellent lodge                6
wonderful lodge                6
nice location                 6
very good lodge                5

So we will filter out all feedback shorter than 20 symbols — 556 out of 12 890 critiques (4.3%). Then, we are going to analyse solely lengthy statements with extra context. It’s an arbitrary threshold primarily based on examples, you possibly can attempt a few ranges and see what texts are filtered out.

It’s value checking whether or not this filter disproportionally impacts some lodges. Shares of quick feedback are fairly shut for various classes. So, the info appears to be like OK.

Now, it’s time to construct our first matter mannequin. Let’s begin easy with essentially the most primary one to know how library works, then we are going to enhance it.

We will practice a subject mannequin in only a few code traces that may very well be simply understood by anybody who has used a minimum of one ML package deal earlier than.

from bertopic import BERTopic
docs = checklist(df.critiques.values)
topic_model = BERTopic()
matters, probs = topic_model.fit_transform(docs)

The default mannequin returned 113 matters. We will have a look at prime matters.

topic_model.get_topic_info().head(7).set_index('Matter')[
['Count', 'Name', 'Representation']]

The most important group is Matter -1 , which corresponds to outliers. By default, BERTopic makes use of HDBSCAN for clustering, and it doesn’t drive all knowledge factors to be a part of clusters. In our case, 6 356 critiques are outliers (round 49.3% of all critiques). It’s virtually a half of our knowledge, so we are going to work with this group later.

A subject illustration is often a set of most necessary phrases particular to this matter and never others. So, one of the simplest ways to know a subject is to have a look at the principle phrases (in BERTopic, a class-based TF-IDF rating is used to rank the phrases).

topic_model.visualize_barchart(top_n_topics = 16, n_words = 10)

BERTopic even has Matters per Class illustration that may remedy our job of understanding the variations in course critiques.

topics_per_class = topic_model.topics_per_class(docs, 
lessons=filt_df.lodge)
topic_model.visualize_topics_per_class(topics_per_class, 
top_n_topics=10, normalize_frequency = True)

In case you are questioning find out how to interpret this graph, you aren’t alone — I additionally wasn’t capable of guess. Nonetheless, the creator kindly helps this package deal, and there are a variety of solutions on GitHub. From the dialogue, I realized that the present normalisation strategy doesn’t present the share of various matters for lessons. So, it hasn’t fully solved our preliminary job.

Nonetheless, we did the primary iteration in lower than 10 rows of code. It’s incredible, however there’s some room for enchancment.

As we noticed earlier, virtually 50% of knowledge factors are thought of outliers. It’s rather a lot, let’s see what we may do with it.

The documentation supplies 4 completely different methods to cope with the outliers:

• primarily based on topic-document possibilities,
• primarily based on matter distributions,
• primarily based on c-TF-IFD representations,
• primarily based on doc and matter embeddings.

You possibly can attempt completely different methods and see which one matches your knowledge one of the best.

Let’s have a look at examples of outliers. Despite the fact that these critiques are comparatively quick, they’ve a number of matters.

BERTopic makes use of clustering to outline matters. It implies that not multiple matter is assigned to every doc. In most real-life circumstances, you possibly can have a mix of matters in your texts. We could also be unable to assign a subject to the paperwork as a result of they’ve a number of ones.

Fortunately, there’s an answer for it — use Matter Distributions. With such an strategy, every doc shall be break up into tokens. Then, we are going to type subsentences (outlined by sliding window and stride) and assign a subject for every such subsentence.

Let’s do this strategy and see whether or not we can cut back the variety of outliers with out matters.

Nonetheless, Matter Distributions are primarily based on the fitted matter mannequin, so let’s improve it.

To start with, we will use CountVectorizer. It defines how a doc shall be break up into tokens. Additionally, it may assist us to do away with meaningless phrases like to, not or the (there are a variety of such phrases in our first mannequin).

Additionally, we may enhance matters’ representations and even attempt a few completely different fashions. I used the KeyBERTInspired mannequin (extra particulars), however you possibly can attempt different choices (for instance, LLMs).

from sklearn.feature_extraction.textual content import CountVectorizer
from bertopic.illustration import KeyBERTInspired, PartOfSpeech, MaximalMarginalRelevance
main_representation_model = KeyBERTInspired()
aspect_representation_model1 = PartOfSpeech("en_core_web_sm")
aspect_representation_model2 = [KeyBERTInspired(top_n_words=30), 
MaximalMarginalRelevance(diversity=.5)]
representation_model = {
"Essential": main_representation_model,
"Aspect1":  aspect_representation_model1,
"Aspect2":  aspect_representation_model2 
}
vectorizer_model = CountVectorizer(min_df=5, stop_words = 'english')
topic_model = BERTopic(nr_topics = 'auto', 
vectorizer_model = vectorizer_model,
representation_model = representation_model)
matters, ini_probs = topic_model.fit_transform(docs)

I specified nr_topics = 'auto' to scale back the variety of matters. Then, all matters with a similarity over threshold shall be merged routinely. With this function, we bought 99 matters.

I’ve created a perform to get prime matters and their shares so we may analyse it simpler. Let’s have a look at the brand new set of matters.

def get_topic_stats(topic_model, extra_cols = []):
topics_info_df = topic_model.get_topic_info().sort_values('Depend', ascending = False)
topics_info_df['Share'] = 100.*topics_info_df['Count']/topics_info_df['Count'].sum()
topics_info_df['CumulativeShare'] = 100.*topics_info_df['Count'].cumsum()/topics_info_df['Count'].sum()
return topics_info_df[['Topic', 'Count', 'Share', 'CumulativeShare', 
'Name', 'Representation'] + extra_cols]
get_topic_stats(topic_model, ['Aspect1', 'Aspect2']).head(10)
.set_index('Matter')

We will additionally have a look at the Interoptic distance map to higher perceive our clusters, for instance, that are shut to one another. You can even use it to outline some mother or father matters and subtopics. It’s referred to as Hierarchical Matter Modelling and you should utilize different instruments for it.

topic_model.visualize_topics()

One other insightful method to higher perceive your matters is to have a look at visualize_documents graph (documentation).

We will see that the variety of matters has diminished considerably. Additionally, there are not any meaningless cease phrases in matters’ representations.

Nonetheless, we nonetheless see comparable matters within the outcomes. We will examine and merge such matters manually.

For this, we will draw a Similarity matrix. I specified n_clusters, and our matters had been clustered to visualise them higher.

topic_model.visualize_heatmap(n_clusters = 20)

There are some fairly shut matters. Let’s calculate the pair distances and have a look at the highest matters.

from sklearn.metrics.pairwise import cosine_similarity
distance_matrix = cosine_similarity(np.array(topic_model.topic_embeddings_))
dist_df = pd.DataFrame(distance_matrix, columns=topic_model.topic_labels_.values(), 
index=topic_model.topic_labels_.values())
tmp = []
for rec in dist_df.reset_index().to_dict('information'):
t1 = rec['index']
for t2 in rec:
if t2 == 'index': 
proceed
tmp.append(
{
'topic1': t1, 
'topic2': t2, 
'distance': rec[t2]
}
)
pair_dist_df = pd.DataFrame(tmp)
pair_dist_df = pair_dist_df[(pair_dist_df.topic1.map(
lambda x: not x.startswith('-1'))) & 
(pair_dist_df.topic2.map(lambda x: not x.startswith('-1')))]
pair_dist_df = pair_dist_df[pair_dist_df.topic1 < pair_dist_df.topic2]
pair_dist_df.sort_values('distance', ascending = False).head(20)

I discovered steering on find out how to get the space matrix from GitHub discussions.

We will now see the highest pairs of matters by cosine similarity. There are matters with shut meanings that we may merge.

topic_model.merge_topics(docs, [[26, 74], [43, 68, 62], [16, 50, 91]])
df['merged_topic'] = topic_model.topics_

Consideration: after merging, all matters’ IDs and representations shall be recalculated, so it’s value updating if you happen to use them.

Now, we’ve improved our preliminary mannequin and are prepared to maneuver on.

With real-life duties, it’s value spending extra time on merging matters and attempting completely different approaches to illustration and clustering to get one of the best outcomes.

The opposite potential concept is splitting critiques into separate sentences as a result of feedback are reasonably lengthy.

Let’s calculate matters’ and tokens’ distributions. I’ve used a window equal to 4 (the creator suggested utilizing 4–8 tokens) and stride equal 1.

topic_distr, topic_token_distr = topic_model.approximate_distribution(
docs, window = 4, calculate_tokens=True)

For instance, this remark shall be break up into subsentences (or units of 4 tokens), and the closest of present matters shall be assigned to every. Then, these matters shall be aggregated to calculate possibilities for the entire sentence. You’ll find extra particulars within the documentation.

Utilizing this knowledge, we will get the possibilities of various matters for every evaluation.

topic_model.visualize_distribution(topic_distr[doc_id], min_probability=0.05)

We will even see the distribution of phrases for every matter and perceive why we bought this consequence. For our sentence, finest very stunningwas the principle time period for Matter 74, whereas location nearoutlined a bunch of location-related matters.

vis_df = topic_model.visualize_approximate_distribution(docs[doc_id], 
topic_token_distr[doc_id])
vis_df

This instance additionally reveals that we would have spent extra time merging matters as a result of there are nonetheless fairly comparable ones.

Now, we have now possibilities for every matter and evaluation. The following job is to pick a threshold to filter irrelevant matters with too low likelihood.

We will do it as normal utilizing knowledge. Let’s calculate the distribution of chosen matters per evaluation for various threshold ranges.

tmp_dfs = []
# iterating via completely different threshold ranges
for thr in tqdm.tqdm(np.arange(0, 0.35, 0.001)):
# calculating variety of matters with likelihood > threshold for every doc
tmp_df = pd.DataFrame(checklist(map(lambda x: len(checklist(filter(lambda y: y >= thr, x))), topic_distr))).rename(
columns = {0: 'num_topics'}
)
tmp_df['num_docs'] = 1
tmp_df['num_topics_group'] = tmp_df['num_topics']
.map(lambda x: str(x) if x < 5 else '5+')
# aggregating stats
tmp_df_aggr = tmp_df.groupby('num_topics_group', as_index = False).num_docs.sum()
tmp_df_aggr['threshold'] = thr
tmp_dfs.append(tmp_df_aggr)
num_topics_stats_df = pd.concat(tmp_dfs).pivot(index = 'threshold', 
values = 'num_docs',
columns = 'num_topics_group').fillna(0)
num_topics_stats_df = num_topics_stats_df.apply(lambda x: 100.*x/num_topics_stats_df.sum(axis = 1))
# visualisation
colormap = px.colours.sequential.YlGnBu
px.space(num_topics_stats_df, 
title = 'Distribution of variety of matters',
labels = {'num_topics_group': 'variety of matters',
'worth': 'share of critiques, %'},
color_discrete_map = {
'0': colormap[0],
'1': colormap[3],
'2': colormap[4],
'3': colormap[5],
'4': colormap[6],
'5+': colormap[7]
})

threshold = 0.05 appears to be like like a superb candidate as a result of, with this stage, the share of critiques with none matter remains to be low sufficient (lower than 6%), whereas the proportion of feedback with 4+ matters can also be not so excessive.

This strategy has helped us to scale back the variety of outliers from 53.4% to five.8%. So, assigning a number of matters may very well be an efficient method to deal with outliers.

Let’s calculate the matters for every doc with this threshold.

threshold = 0.13
# outline matter with likelihood > 0.13 for every doc
df['multiple_topics'] = checklist(map(
lambda doc_topic_distr: checklist(map(
lambda y: y[0], filter(lambda x: x[1] >= threshold, 
(enumerate(doc_topic_distr)))
)), topic_distr
))
# making a dataset with docid, matter
tmp_data = []
for rec in df.to_dict('information'):
if len(rec['multiple_topics']) != 0:
mult_topics = rec['multiple_topics']
else:
mult_topics = [-1]
for matter in mult_topics: 
tmp_data.append(
{
'matter': matter,
'id': rec['id'],
'course_id': rec['course_id'],
'reviews_transl': rec['reviews_transl']
}
)
mult_topics_df = pd.DataFrame(tmp_data)

Now, we have now a number of matters mapped to every evaluation and we will examine matters’ mixtures for various lodge chains.

Let’s discover circumstances when a subject has too excessive or low share for a selected lodge. For that, we are going to calculate for every pair matter + lodge share of feedback associated to the subject for this lodge vs. all others.

tmp_data = []
for lodge in mult_topics_df.lodge.distinctive():
for matter in mult_topics_df.matter.distinctive():
tmp_data.append({
'lodge': lodge,
'topic_id': matter,
'total_hotel_reviews': mult_topics_df[mult_topics_df.hotel == hotel].id.nunique(),
'topic_hotel_reviews': mult_topics_df[(mult_topics_df.hotel == hotel) 
& (mult_topics_df.topic == topic)].id.nunique(),
'other_hotels_reviews': mult_topics_df[mult_topics_df.hotel != hotel].id.nunique(),
'topic_other_hotels_reviews': mult_topics_df[(mult_topics_df.hotel != hotel) 
& (mult_topics_df.topic == topic)].id.nunique()
})
mult_topics_stats_df = pd.DataFrame(tmp_data)
mult_topics_stats_df['topic_hotel_share'] = 100*mult_topics_stats_df.topic_hotel_reviews/mult_topics_stats_df.total_hotel_reviews
mult_topics_stats_df['topic_other_hotels_share'] = 100*mult_topics_stats_df.topic_other_hotels_reviews/mult_topics_stats_df.other_hotels_reviews

Nonetheless, not all variations are important for us. We will say that the distinction in matters’ distribution is value if there are

• statistical significance — the distinction isn’t just by likelihood,
• sensible significance — the distinction is greater than X% factors (I used 1%).

from statsmodels.stats.proportion import proportions_ztest
mult_topics_stats_df['difference_pval'] = checklist(map(
lambda x1, x2, n1, n2: proportions_ztest(
depend = [x1, x2],
nobs = [n1, n2],
different = 'two-sided'
)[1],
mult_topics_stats_df.topic_other_hotels_reviews,
mult_topics_stats_df.topic_hotel_reviews,
mult_topics_stats_df.other_hotels_reviews,
mult_topics_stats_df.total_hotel_reviews
))
mult_topics_stats_df['sign_difference'] = mult_topics_stats_df.difference_pval.map(
lambda x: 1 if x <= 0.05 else 0
)
def get_significance(d, signal):
sign_percent = 1
if signal == 0:
return 'no diff'
if (d >= -sign_percent) and (d <= sign_percent):
return 'no diff'
if d < -sign_percent:
return 'decrease'
if d > sign_percent:
return 'greater'
mult_topics_stats_df['diff_significance_total'] = checklist(map(
get_significance,
mult_topics_stats_df.topic_hotel_share - mult_topics_stats_df.topic_other_hotels_share,
mult_topics_stats_df.sign_difference
))

Now we have all of the stats for all matters and lodges, and the final step is to create a visualisation evaluating matter shares by classes.

import plotly
# outline colour relying on distinction significance
def get_color_sign(rel):
if rel == 'no diff':
return plotly.colours.qualitative.Set2[7]
if rel == 'decrease':
return plotly.colours.qualitative.Set2[1]
if rel == 'greater':
return plotly.colours.qualitative.Set2[0]
# return matter illustration in an appropriate for graph title format
def get_topic_representation_title(topic_model, matter):
knowledge = topic_model.get_topic(matter)
knowledge = checklist(map(lambda x: x[0], knowledge))
return ', '.be part of(knowledge[:5]) + ', <br>         ' + ', '.be part of(knowledge[5:])
def get_graphs_for_topic(t):
topic_stats_df = mult_topics_stats_df[mult_topics_stats_df.topic_id == t]
.sort_values('total_hotel_reviews', ascending = False).set_index('lodge')
colours = checklist(map(
get_color_sign,
topic_stats_df.diff_significance_total
))
fig = px.bar(topic_stats_df.reset_index(), x = 'lodge', y = 'topic_hotel_share',
title = 'Matter: %s' % get_topic_representation_title(topic_model, 
topic_stats_df.topic_id.min()),
text_auto = '.1f',
labels = {'topic_hotel_share': 'share of critiques, %'},
hover_data=['topic_id'])
fig.update_layout(showlegend = False)
fig.update_traces(marker_color=colours, marker_line_color=colours,
marker_line_width=1.5, opacity=0.9)
topic_total_share = 100.*((topic_stats_df.topic_hotel_reviews + topic_stats_df.topic_other_hotels_reviews)
/(topic_stats_df.total_hotel_reviews + topic_stats_df.other_hotels_reviews)).min()
print(topic_total_share)
fig.add_shape(kind="line",
xref="paper",
x0=0, y0=topic_total_share,
x1=1, y1=topic_total_share,
line=dict(
colour=colormap[8],
width=3, sprint="dot"
)
)
fig.present()

Then, we will calculate the highest matters checklist and make graphs for them.

top_mult_topics_df = mult_topics_df.groupby('matter', as_index = False).id.nunique()
top_mult_topics_df['share'] = 100.*top_mult_topics_df.id/top_mult_topics_df.id.sum()
top_mult_topics_df['topic_repr'] = top_mult_topics_df.matter.map(
lambda x: get_topic_representation(topic_model, x)
)
for t in top_mult_topics_df.head(32).matter.values:
get_graphs_for_topic(t)

Listed here are a few examples of ensuing charts. Let’s attempt to make some conclusions primarily based on this knowledge.

We will see that Vacation Inn, Travelodge and Park Inn have higher costs and worth for cash in comparison with Hilton or Park Plaza.

The opposite perception is that in Travelodge noise could also be an issue.

It’s a bit difficult for me to interpret this consequence. I’m unsure what this matter is about.

One of the best observe for such circumstances is to have a look at some examples.

• We stayed within the East tower the place the lifts are below renovation, just one works, however there are indicators exhibiting the best way to service lifts which can be utilized additionally.
• Nonetheless, the carpet and the furnishings may have a refurbishment.
• It’s constructed proper over Queensway station. Beware that this tube cease shall be closed for refurbishing for one 12 months! So that you may think about noise ranges.

So, this matter is in regards to the circumstances of non permanent points in the course of the lodge keep or furnishings not in one of the best situation.

You’ll find the complete code on GitHub.

At this time, we’ve performed an end-to-end Matter Modelling evaluation:

• Construct a primary matter mannequin utilizing the BERTopic library.
• Then, we’ve dealt with outliers, so solely 5.8% of our critiques don’t have a subject assigned.
• Decreased the variety of matters each routinely and manually to have a concise checklist.
• Discovered find out how to assign a number of matters to every doc as a result of, most often, your textual content can have a mix of matters.

Lastly, we had been capable of examine critiques for various programs, create inspiring graphs and get some insights.

Thank you numerous for studying this text. I hope it was insightful to you. If in case you have any follow-up questions or feedback, please depart them within the feedback part.

Ganesan, Kavita and Zhai, ChengXiang. (2011). OpinRank Evaluation Dataset.
UCI Machine Studying Repository. https://doi.org/10.24432/C5QW4W