Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to play on Kaggle". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to play on Kaggle.

Introduction

Kaggle is the most famous machine learning competition website. The Kaggle contest consists of a dataset that is available from the website and requires the use of machines, deep learning, or other data science and technology to solve problems. Once you have found a solution, you can upload the results of your model to the website, and the site will rank you according to your results. If your results can beat other contestants, then you may get a cash reward.

Kaggle is a great place to hone your machine learning and data science skills, where you can compare yourself with others and learn new technologies.

Twitter data set

Kaggle's latest competition offers a dataset of tweets and a hashtag that tells us whether these tweets are really about disasters. There are nearly 3000 contestants in the ranking of the competition, with a maximum prize of 10, 000 US dollars.

If you don't already have a Kaggle account, you can create one for free.

If you select "download all" from the competition page, you will get a zip file containing three CSV files:

The first data file, train.csv, contains a set of features and their corresponding target tags for training purposes. The dataset consists of the following attributes:

Id: the numeric identifier of the tweet. This will be very important when we upload our predictions to the rankings.

Keyword: a keyword in a tweet that may not be available in some cases.

Location: where the tweet is sent, which may not exist.

Text: the full text of a tweet.

Goal: this is the label we are trying to predict. If this tweet is really about a disaster, it will be 1, if not, it will be 0.

Let's learn more about this. In the following code, you will notice that I used a set_option command. This command from the Pandas library allows you to control the format in which dataframe results are displayed. I use this command here to ensure that the full contents of the text column are displayed, which makes my results and analysis easier to view:

Import pandas as pdpd.set_option ('display.max_colwidth',-1) train_data = pd.read_csv (' train.csv') train_data.head ()

The second data file, test.csv, is a test set that contains only features and no tags. For this dataset, we will predict the target label and use the results to get a place in the rankings.

Test_data = pd.read_csv ('test.csv') test_data.head ()

The third file, sample_submission, is an example that shows what the submission looks like. This file will contain the id column in the test.csv file and the target we predicted with the model. Once we have created this file, we will submit it to the website and get a ranking of the location.

Sample_submission = pd.read_csv ('sample_submission.csv') sample_submission.head () data cleanup

For any machine learning task, we must perform some data cleaning and preprocessing before we can train a model. This is especially important when working with text data.

To simplify our first model, and because there is a lot of missing data in these columns, we will remove the location and keyword features and train only with the actual text from tweet. We will also delete the id column because it is of no use to the training model.

Train_data = train_data.drop (['keyword',' location', 'id'], axis=1) train_data.head ()

Our dataset looks like this:

Text often contains many special characters that are not necessarily meaningful for machine learning algorithms. So the first step I'm going to take is to delete these. I also lowercase all the words.

Import redef clean_text (df, text_field): DF [text _ field] = DF [text _ field] .str.lower () DF [text _ field] = DF [text _ field] .apply (lambda elem: re.sub (r "(@ [text-9] +) | ([^ 0-9A-Za-z\ t]) | (\ text:\ /\ S+) | ^ rt | http.+?", "", elem) return dfdata_clean = clean_text (train_data) "text") data_clean.head ()

Another useful text cleanup process is to delete the stop word. The stop word is a very common word that usually conveys little meaning. In English, these words include "the", "it" and "as". If we leave these words in the text, they will make a lot of noise, which will make the algorithm more difficult to learn.

NLTK is a collection of python libraries and tools for working with text data. In addition to the processing tools, NLTK also has a large text corpus and vocabulary resources, including all stop words in various languages. We will use this library to remove the stop word from the dataset.

The NLTK library can be installed through pip. After installation, you need to import the library collection and download the stopwords file:

Import nltk.corpusnltk.download ('stopwords')

Once this is done, you can read the stop word and use it to delete their tweets.

From nltk.corpus import stopwordsstop = stopwords.words ('english') data_clean [' text'] = data_clean ['text'] .apply (lambda x:' .join ([word for word in x.split () if word not in (stop)])) data_clean.head () data preprocessing

Once the data is cleaned up, further preprocessing is needed to prepare for the use of machine learning algorithms.

All machine learning algorithms use mathematical calculations to map features (text or words in our case) and patterns in target variables. Therefore, before training the machine learning model, the text must be converted into a numerical representation in order to carry out these calculations.

There are many methods for this type of preprocessing, but in this example, I will use two methods from the scikit-learn library.

The first step in this process is to split the data into tags or individual words, calculate how often each word appears in the text, and then represent these counts as a sparse matrix. The CountVectoriser function can do this.

The next step is to weight the number of words generated by CountVectoriser. The purpose of applying this weighting is to reduce the influence of words with very high frequency in the text, so that words with low frequency and possibly large amount of information are considered to be very important in model training. TfidTransformer can perform this function.

Machine learning process

Let's put all of this preprocessing and model fitting into the scikit-learn process to see how the model is executed. For my first attempt, I used the linear support vector machine classifier (SGDClassifier) because it is generally considered to be one of the best text classification algorithms.

From sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split (data_clean ['text'], data_clean [' target'], random_state = 0) from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.pipeline import Pipelinefrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfTransformerfrom sklearn.linear_model import SGDClassifierpipeline_sgd = Pipeline ([('vect', CountVectorizer ()), (' tfidf', TfidfTransformer () ('nb', SGDClassifier ()),]) model = pipeline_sgd.fit (X_train, y_train)

Let's use this trained model to predict our test data and see how the model works.

From sklearn.metrics import classification_reporty_predict = model.predict (X_test) print (classification_report (y_test, y_predict))

For the first attempt, the model performed fairly well.

Submit results

Now let's take a look at the performance of this model on the competitive test dataset and our ranking in the rankings.

First, we need to clear the text in the test file and use the model to predict. The following code takes a copy of the test data and performs the same cleanup we applied to the training data. The output is shown in the following code.

Submission_test_clean = test_data.copy () submission_test_clean = clean_text (submission_test_clean, "text") submission_test_clean ['text'] = submission_test_clean [' text'] .apply (lambda x: '.join ([word for word in x.split () if word not in (stop)])) submission_test_clean = submission_test_clean [' text'] submission_test_clean.head ()

Next, we use the model to create predictions.

Submission_test_pred = model.predict (submission_test_clean)

To create a submission, we need to construct a dataframe that contains only the id from the test set and our predictions.

Id_col = test_data ['id'] submission_df_1 = pd.DataFrame ({"id": id_col, "target": submission_test_pred}) submission_df_1.head ()

Finally, we save it as a CSV file. Index=False must be included, or the index will be saved as a column in the file and your submission will be rejected.

Submission_df_1.to_csv ('submission_1.csv', index=False)

Once we have the CSV file, we can return to the match page and select the submit Forecast button. This opens a form where you can upload the CSV file. It's a good idea to add some comments about the method so that you have a record of previous submission attempts.

After submitting the file, you will see the following results:

Now we have a successful submission!

This model gave me a score of 0.78 and ranked 2371 in the ranking. Obviously there is some room for improvement, but now I already have a benchmark for future submission.

Thank you for your reading, the above is the content of "how to play on Kaggle". After the study of this article, I believe you have a deeper understanding of how to play on Kaggle, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.