On this tutorial, you’ll discover ways to use Python and the OpenAI API to carry out information mining and evaluation in your information.
Manually analyzing datasets to extract helpful information, and even utilizing easy applications to do the identical, can typically get sophisticated and time consuming. Fortunately, with the OpenAI API and Python it’s attainable to systematically analyze your datasets for attention-grabbing data with out over-engineering your code and losing time. This can be utilized as a common answer for information evaluation, eliminating the necessity to use totally different strategies, libraries and APIs to research several types of information and information factors inside a dataset.
Let’s stroll via the steps of utilizing the OpenAI API and Python to research your information, beginning with learn how to set issues up.
Setup
To mine and analyze information via Python utilizing the OpenAI API, set up the openai and pandas libraries:
pip3 set up openai pandas
After you’ve performed that, create a brand new folder and create an empty Python file inside your new folder.
Analyzing Textual content Recordsdata
For this tutorial, I believed it will be attention-grabbing to make Python analyze Nvidia’s newest earnings name.
Obtain the most recent Nvidia earnings name transcript that I obtained from The Motley Idiot and transfer it into your challenge folder.
Then open your empty Python file and add this code.
The code reads the Nvidia earnings transcript that you simply’ve downloaded and passes it to the extract_info perform because the transcript variable.
The extract_info perform passes the immediate and transcript because the person enter, in addition to temperature=0.3 and mannequin="gpt-3.5-turbo-16k". The explanation it makes use of the “gpt-3.5-turbo-16k” mannequin is as a result of it may course of massive texts comparable to this transcript. The code will get the response utilizing the openai.ChatCompletion.create endpoint and passes the immediate and transcript variables as person enter:
completions = openai.ChatCompletion.create(
mannequin="gpt-3.5-turbo-16k",
messages=[
{"role": "user", "content": prompt+"nn"+text}
],
temperature=0.3,
)
The total enter will appear like this:
Extract the next data from the textual content:
Nvidia's income
What Nvidia did this quarter
Remarks about AI
Nvidia earnings transcript goes right here
Now, if we go the enter to the openai.ChatCompletion.create endpoint, the total output will appear like this:
{
"decisions": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Actual response",
"role": "assistant"
}
}
],
"created": 1693336390,
"id": "request-id",
"mannequin": "gpt-3.5-turbo-16k-0613",
"object": "chat.completion",
"utilization": {
"completion_tokens": 579,
"prompt_tokens": 3615,
"total_tokens": 4194
}
}
As you may see, it returns the textual content response in addition to the token utilization of the request, which may be helpful for those who’re monitoring your bills and optimizing your prices. However since we’re solely within the response textual content, we get it by specifying the completions.decisions[0].message.content material response path.
For those who run your code, you must get an identical output to what’s quoted beneath:
From the textual content, we are able to extract the next data:
- Nvidia’s income: Within the second quarter of fiscal 2024, Nvidia reported report Q2 income of 13.51 billion, which was up 88% sequentially and up 101% yr on yr.
- What Nvidia did this quarter: Nvidia skilled distinctive development in varied areas. They noticed report income of their information heart section, which was up 141% sequentially and up 171% yr on yr. Additionally they noticed development of their gaming section, with income up 11% sequentially and 22% yr on yr. Moreover, their skilled visualization section noticed income development of 28% sequentially. Additionally they introduced partnerships and collaborations with corporations like Snowflake, ServiceNow, Accenture, Hugging Face, VMware, and SoftBank.
- Remarks about AI: Nvidia highlighted the robust demand for his or her AI platforms and accelerated computing options. They talked about the deployment of their HGX techniques by main cloud service suppliers and client web corporations. Additionally they mentioned the purposes of generative AI in varied industries, comparable to advertising, media, and leisure. Nvidia emphasised the potential of generative AI to create new market alternatives and increase productiveness in several sectors.
As you may see, the code extracts the information that’s specified within the immediate (Nvidia’s income, what Nvidia did this quarter, and remarks about AI) and prints it.
Analyzing CSV Recordsdata
Analyzing earnings-call transcripts and textual content information is cool, however to systematically analyze massive volumes of information, you’ll have to work with CSV information.
As a working instance, obtain this Medium articles CSV dataset and paste it into your challenge file.
For those who have a look into the CSV file, you’ll see that it has the “creator”, “claps”, “reading_time”, “hyperlink”, “title” and “textual content” columns. For analyzing the medium articles with OpenAI, you solely want the “title” and “textual content” columns.
Create a brand new Python file in your challenge folder and paste this code.
This code is a bit totally different from the code we used to research a textual content file. It reads CSV rows one after the other, extracts the desired items of data, and provides them into new columns.
For this tutorial, I’ve picked a CSV dataset of Medium articles, which I obtained from HSANKESARA on Kaggle. This CSV evaluation code will discover the general tone and the primary lesson/level of every article, utilizing the “title” and “article” columns of the CSV file. Since I all the time come throughout clickbaity articles on Medium, I additionally thought it will be attention-grabbing to inform it to search out how “clickbaity” every article is by giving every one a “clickbait rating” from 0 to three, the place 0 is not any clickbait and three is excessive clickbait.
Earlier than I clarify the code, analyzing the whole CSV file would take too lengthy and price too many API credit, so for this tutorial, I’ve made the code analyze solely the primary 5 articles utilizing df = df[:5].
It’s possible you’ll be confused in regards to the following a part of the code, so let me clarify:
for di in vary(len(df)):
title = titles[di]
summary = articles[di]
additional_params = extract_info('Title: '+str(title) + 'nn' + 'Textual content: ' + str(summary))
strive:
end result = additional_params.cut up("nn")
besides:
end result = {}
This code iterates via all of the articles (rows) within the CSV file and, with every iteration, will get the title and physique of every article and passes it to the extract_info perform, which we noticed earlier. It then turns the response of the extract_info perform into an inventory to separate the totally different items of information utilizing this code:
strive:
end result = additional_params.cut up("nn")
besides:
end result = {}
Subsequent, it provides each bit of information into an inventory, and if there’s an error (if there’s no worth), it provides “No end result” into the listing:
strive:
apa1.append(end result[0])
besides Exception as e:
apa1.append('No end result')
strive:
apa2.append(end result[1])
besides Exception as e:
apa2.append('No end result')
strive:
apa3.append(end result[2])
besides Exception as e:
apa3.append('No end result')
Lastly, after the for loop is completed, the lists that comprise the extracted data are inserted into new columns within the CSV file:
df = df.assign(Tone=apa1)
df = df.assign(Main_lesson_or_point=apa2)
df = df.assign(Clickbait_score=apa3)
As you may see, it provides the lists into new CSV columns which are identify “Tone”, “Main_lesson_or_point” and “Clickbait_score”.
It then appends them to the CSV file with index=False:
df.to_csv("information.csv", index=False)
The explanation why you must specify index=False is to keep away from creating new index columns each time you append new columns to the CSV file.
Now, for those who run your Python file, look ahead to it to complete and verify our CSV file in a CSV file viewer, you’ll see the brand new columns, as pictured beneath.
For those who run your code a number of instances, you’ll discover that the generated solutions differ barely. It is because the code makes use of temperature=0.3 so as to add a little bit of creativity into its solutions, which is beneficial for subjective matters like clickbait.
Working with A number of Recordsdata
If you wish to routinely analyze a number of information, you could first put them inside a folder and ensure the folder solely comprises the information you’re fascinated about, to forestall your Python code from studying irrelevant information. Then, set up the glob library utilizing pip3 set up glob and import it in your Python file utilizing import glob.
In your Python file, use this code to get an inventory of all of the information in your information folder:
data_files = glob.glob("data_folder/*")
Then put the code that does the evaluation in a for loop:
for i in vary(len(data_files)):
Contained in the for loop, learn the contents of every file like this for textual content information:
f = open(f"data_folder/{data_files[i]}", "r")
txt_data = f.learn()
Additionally like this for CSV information:
df = pd.read_csv(f"data_folder/{data_files[i]}")
As well as, make sure that to save lots of the output of every file evaluation right into a separate file utilizing one thing like this:
df.to_csv(f"output_folder/information{i}.csv", index=False)
Conclusion
Bear in mind to experiment together with your temperature parameter and regulate it to your use case. If you would like the AI to make extra artistic solutions, enhance your temperature, and if you need it to make extra factual solutions, make sure that to decrease it.
The mix of OpenAI and Python information evaluation has many purposes aside from article and earnings name transcript evaluation. Examples embody information evaluation, e-book evaluation, buyer evaluate evaluation, and way more! That stated, when testing your Python code on huge datasets, make sure that to solely take a look at it on a small a part of the total dataset to save lots of API credit and time.
