Extracting blocks of information from a txt file and creating a Pandas frame and storage
I have a large txt file that includes chat transcripts. My goal is to extract the various components and create a Pandas Df to store in it. A sample chat looks like this:
*****************************************************
Session:123456
Chat Date: 2017-05-01T08:01:45+00:00
Chat exec name: Sam
Member name: Sara
2017-05-01T08:01:45+00:00 Sara: I need help on element A
2017-05-01T08:01:47+00:00 Sam: Sure I can help you on this one
2017-05-01T08:01:48+00:00 Sara: Is there a better product
2017-05-01T08:01:48+10:00 Sam: Sure we have a lot of new products
2017-05-01T08:01:49+18:00 Sara: Can you let me know
2017-05-01T08:01:51+20:00 Sam: Here is the solution
2017-05-01T08:01:52+00:00 Sara: Thanks for this
2017-05-01T08:01:52+11:00 Sam: Have a Nive day Bye!!
*****************************************************
Session:234567
Chat Date: 2017-05-02T18:00:30+00:00
Chat exec name: PAUL
Member name:CHRIS
2017-05-02T18:00:30+00:00 CHRIS: I need help on element A
2017-05-02T18:02:30+00:00 PAUL: Sure I can help you on this one
2017-05-02T18:02:39+00:00 CHRIS: Is there a better product
2017-05-02T18:04:01+00:00 PAUL: Sure we have a lot of new products
2017-05-02T18:04:30+00:00 CHRIS: Can you let me know
2017-05-02T18:08:11+00:00 PAUL: Here is the solution
2017-05-02T18:08:59+00:00 CHRIS: Thanks for this
2017-05-02T18:09:11+00:00 PAUL: Have a Nice day Bye!!
*****************************************************
If I can create a table with columns:
Session, ChatDate, ChatExecName, member name, time, person, sentence
The first four columns must be repeated for a complete chat block. moreover, the delimiters are fixed and never change.
I tried this but this is returning all blocks together, can someone help.
import re
def GetTheSentences(infile):
Delim1 = '*****************************************************'
Delim2 = '*****************************************************'
with open(infile) as fp:
for result in re.findall('Delim1(.*?)Delim2', fp.read(), re.S):
print (result)
and
import re
def GetTheSentences2(file):
start_rx =re.compile('*****************************************************')
end_rx = re.compile('*****************************************************')
start = False
output = []
with open(file, encoding="latin-1") as datafile:
for line in datafile.readlines():
if re.match(start_rx, line):
start = True
elif re.match(end_rx, line):
start = False
if start:
output.append(line)
print (output)
+3
source to share
1 answer
I hope this is helpful:
data = '''*****************************************************
Session:123456
Chat Date: 2017-05-01T08:01:45+00:00
Chat exec name: Sam
Member name: Sara
2017-05-01T08:01:45+00:00 Sara: I need help on element A
2017-05-01T08:01:47+00:00 Sam: Sure I can help you on this one
2017-05-01T08:01:48+00:00 Sara: Is there a better product
2017-05-01T08:01:48+10:00 Sam: Sure we have a lot of new products
2017-05-01T08:01:49+18:00 Sara: Can you let me know
2017-05-01T08:01:51+20:00 Sam: Here is the solution
2017-05-01T08:01:52+00:00 Sara: Thanks for this
2017-05-01T08:01:52+11:00 Sam: Have a Nive day Bye!!
*****************************************************
Session:234567
Chat Date: 2017-05-02T18:00:30+00:00
Chat exec name: PAUL
Member name:CHRIS
2017-05-02T18:00:30+00:00 CHRIS: I need help on element A
2017-05-02T18:02:30+00:00 PAUL: Sure I can help you on this one
2017-05-02T18:02:39+00:00 CHRIS: Is there a better product
2017-05-02T18:04:01+00:00 PAUL: Sure we have a lot of new products
2017-05-02T18:04:30+00:00 CHRIS: Can you let me know
2017-05-02T18:08:11+00:00 PAUL: Here is the solution
2017-05-02T18:08:59+00:00 CHRIS: Thanks for this
2017-05-02T18:09:11+00:00 PAUL: Have a Nice day Bye!!
*****************************************************'''
data = data.split('*****************************************************')
data = [item.split('\n') for item in data if item]
result = []
for group in data:
group = [item for item in group if item]
times = []
people = []
lines = []
for item in group:
if item.startswith('Session'):
session = item.split(':')[-1]
print session
elif item.startswith('Chat Date'):
chatDate = item.split(':', 1)[-1]
elif item.startswith('Chat exec'):
execName = item.split(':')[-1]
elif item.startswith('Member'):
memberName = item.split(':')[-1]
else:
times.append(item[:25])
people.append(item[26:].split(':')[0])
lines.append(item[26:].split(':')[-1])
for i in range(len(times)):
result.append([session, chatDate, execName, memberName, times[i], people[i], lines[i]])
import pandas as pd
df = pd.DataFrame(result, columns=['Session', 'ChatDate', 'ChatExecName', 'Membername', 'Time', 'Person', 'Sentence'])
print df
0
source to share