Automatic Music Generation - Revision history

Techbot: /* Import libraries */

2021-08-28T21:29:51Z

Import libraries

← Older revision		Revision as of 23:29, 28 August 2021
Line 273:		Line 273:

	return np.array(notes)		return np.array(notes)
			</nowiki>
	view rawmusic_1.py hosted with ❤ by GitHub		view rawmusic_1.py hosted with ❤ by GitHub
	~~</nowiki>~~


Line 331:		Line 332:
	</nowiki>		</nowiki>
	view rawmusic_4.py hosted with ❤ by GitHub		view rawmusic_4.py hosted with ❤ by GitHub


	=Output: =		=Output: =

Techbot at 21:28, 28 August 2021

2021-08-28T21:28:46Z

← Older revision		Revision as of 23:28, 28 August 2021
Line 114:		Line 114:
	Delete the first element and pass as an input for the next iteration		Delete the first element and pass as an input for the next iteration
	Repeat steps 2 and 4 for a certain number of iterations		Repeat steps 2 and 4 for a certain number of iterations


	= Understanding the WaveNet Architecture =		= Understanding the WaveNet Architecture =
	The building blocks of WaveNet are Causal Dilated 1D Convolution layers. Let us first understand the importance of the related concepts.		The building blocks of WaveNet are Causal Dilated 1D Convolution layers. Let us first understand the importance of the related concepts.



	= Why and What is a Convolution? =		= Why and What is a Convolution? =
Line 418:		Line 415:
	I have defined 2 architectures here – WaveNet and LSTM. Please experiment with both the architectures to understand the importance of WaveNet architecture.		I have defined 2 architectures here – WaveNet and LSTM. Please experiment with both the architectures to understand the importance of WaveNet architecture.

	def lstm():		<nowiki>def lstm():
	model = Sequential()		model = Sequential()
	model.add(LSTM(128,return_sequences=True))		model.add(LSTM(128,return_sequences=True))
Line 428:		Line 425:
	model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')		model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
	return model		return model
			</nowiki>
	view rawlstm.py hosted with ❤ by GitHub		view rawlstm.py hosted with ❤ by GitHub

Line 433:		Line 431:
	I have simplified the architecture of the WaveNet without adding residual and skip connections since the role of these layers is to improve the faster convergence (and WaveNet takes raw audio wave as input). But in our case, the input would be a set of nodes and chords since we are generating music:		I have simplified the architecture of the WaveNet without adding residual and skip connections since the role of these layers is to improve the faster convergence (and WaveNet takes raw audio wave as input). But in our case, the input would be a set of nodes and chords since we are generating music:

	from keras.layers import *		<nowiki>from keras.layers import *
	from keras.models import *		from keras.models import *
	from keras.callbacks import *		from keras.callbacks import *
Line 465:		Line 463:

	model.summary()		model.summary()
			</nowiki>
	view raw10_8.py hosted with ❤ by GitHub		view raw10_8.py hosted with ❤ by GitHub
	Define the callback to save the best model during training:		Define the callback to save the best model during training:

	mc=ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True,verbose=1)		<nowiki>mc=ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', save_best_only=True,verbose=1)
	Let’s train the model with a batch size of 128 for 50 epochs:		Let’s train the model with a batch size of 128 for 50 epochs:

Line 497:		Line 496:

	print(predictions)		print(predictions)
			</nowiki>
	view raw10_9.py hosted with ❤ by GitHub		view raw10_9.py hosted with ❤ by GitHub
	Now, we will convert the integers back into the notes.		Now, we will convert the integers back into the notes.

	x_int_to_note = dict((number, note_) for number, note_ in enumerate(unique_x))		<nowiki>x_int_to_note = dict((number, note_) for number, note_ in enumerate(unique_x))
	predicted_notes = [x_int_to_note[i] for i in predictions]		predicted_notes = [x_int_to_note[i] for i in predictions]
	The final step is to convert back the predictions into a MIDI file. Let’s define the function to accomplish the task.		The final step is to convert back the predictions into a MIDI file. Let’s define the function to accomplish the task.
Line 539:		Line 539:
	midi_stream = stream.Stream(output_notes)		midi_stream = stream.Stream(output_notes)
	midi_stream.write('midi', fp='music.mid')		midi_stream.write('midi', fp='music.mid')
			</nowiki>
	view raw10_10.py hosted with ❤ by GitHub		view raw10_10.py hosted with ❤ by GitHub
	Converting the predictions into a musical file:		Converting the predictions into a musical file:
Line 546:		Line 547:

	Audio Player		Audio Player

	~~00:00~~
	~~00:00~~

	~~Use Up/Down Arrow keys to increase or decrease volume.~~
	~~Audio Player~~

	~~00:00~~
	~~00:00~~

	~~Use Up/Down Arrow keys to increase or decrease volume.~~
	~~Audio Player~~

	~~00:00~~
	~~00:00~~

	~~Use Up/Down Arrow keys to increase or decrease volume.~~
	~~Awesome, right? But your learning doesn’t stop here. Just remember that we have built a baseline model. There are plenty of ways to improve the performance of the model even further:~~

	~~As the size of the training dataset is small, we can fine-tune a pre-trained model to build a robust system~~
	~~Collect as much as training data as you can since the deep learning model generalizes well on the larger datasets~~


	~~End Notes~~
	Deep Learning has a wide range of applications in our daily life. The key steps in solving any problem are understanding the problem statement, formulating it and defining the architecture to solve the problem.

	~~I had a lot of fun (and learning) while working on this project. Music is a passion of mine and it was quite intriguing combining deep learning with that.~~

	I am looking forward to hearing your approach to the problem in the comments section. And if you have any feedback on this article or any doubts/queries, kindly share them in the comments section below and I will get back to you.

Techbot at 21:24, 28 August 2021

2021-08-28T21:24:12Z

← Older revision		Revision as of 23:24, 28 August 2021
Line 242:		Line 242:
	Let’s define a function straight away for reading the MIDI files. It returns the array of notes and chords present in the musical file.		Let’s define a function straight away for reading the MIDI files. It returns the array of notes and chords present in the musical file.

	<nowiki>#defining function to read MIDI files		<nowiki>#defining function to read MIDI files
	def read_midi(file):		def read_midi(file):

Line 282:		Line 282:
	Now, we will load the MIDI files into our environment		Now, we will load the MIDI files into our environment

	#for listing down the file names		<nowiki>#for listing down the file names
	import os		import os

Line 296:		Line 296:
	#reading each midi file		#reading each midi file
	notes_array = np.array([read_midi(path+i) for i in files])		notes_array = np.array([read_midi(path+i) for i in files])
			</nowiki>
	view rawmusic_2.py hosted with ❤ by GitHub		view rawmusic_2.py hosted with ❤ by GitHub
	Understanding the data:		Understanding the data:
Line 301:		Line 302:
	Under this section, we will explore the dataset and understand it in detail.		Under this section, we will explore the dataset and understand it in detail.

			<nowiki>
	#converting 2D array into 1D array		#converting 2D array into 1D array
	notes_ = [element for note_ in notes_array for element in note_]		notes_ = [element for note_ in notes_array for element in note_]
Line 307:		Line 309:
	unique_notes = list(set(notes_))		unique_notes = list(set(notes_))
	print(len(unique_notes))		print(len(unique_notes))
			</nowiki>
	view rawmusic_3.py hosted with ❤ by GitHub		view rawmusic_3.py hosted with ❤ by GitHub
	Output: 304		Output: 304

	As you can see here, no. of unique notes is 304. Now, let us see the distribution of the notes.		As you can see here, no. of unique notes is 304. Now, let us see the distribution of the notes.
			<nowiki>
	#importing library		#importing library
	from collections import Counter		from collections import Counter
Line 329:		Line 332:
	#plot		#plot
	plt.hist(no)		plt.hist(no)
			</nowiki>
	view rawmusic_4.py hosted with ❤ by GitHub		view rawmusic_4.py hosted with ❤ by GitHub


	Output:		=Output: =

	music generation		music generation



	From the above plot, we can infer that most of the notes have a very low frequency. So, let us keep the top frequent notes and ignore the low-frequency ones. Here, I am defining the threshold as 50. Nevertheless, the parameter can be changed.		From the above plot, we can infer that most of the notes have a very low frequency. So, let us keep the top frequent notes and ignore the low-frequency ones. Here, I am defining the threshold as 50. Nevertheless, the parameter can be changed.

	frequent_notes = [note_ for note_, count in freq.items() if count>=50]		<nowiki>frequent_notes = [note_ for note_, count in freq.items() if count>=50]
	print(len(frequent_notes))		print(len(frequent_notes))
	Output: 167		Output: 167
Line 356:		Line 359:

	new_music = np.array(new_music)		new_music = np.array(new_music)
			</nowiki>
	view rawmusic_5.py hosted with ❤ by GitHub		view rawmusic_5.py hosted with ❤ by GitHub
	Preparing Data:		= Preparing Data =

	Preparing the input and output sequences as mentioned in the article:		Preparing the input and output sequences as mentioned in the article:

	no_of_timesteps = 32		<nowiki>no_of_timesteps = 32
	x = []		x = []
	y = []		y = []
Line 377:		Line 381:
	x=np.array(x)		x=np.array(x)
	y=np.array(y)		y=np.array(y)
			</nowiki>
	view rawmusic_6.py hosted with ❤ by GitHub		view rawmusic_6.py hosted with ❤ by GitHub

Line 382:		Line 387:
	Now, we will assign a unique integer to every note:		Now, we will assign a unique integer to every note:

	unique_x = list(set(x.ravel()))		<nowiki>unique_x = list(set(x.ravel()))
	x_note_to_int = dict((note_, number) for number, note_ in enumerate(unique_x))		x_note_to_int = dict((note_, number) for number, note_ in enumerate(unique_x))
	We will prepare the integer sequences for input data		We will prepare the integer sequences for input data
Line 396:		Line 401:

	x_seq = np.array(x_seq)		x_seq = np.array(x_seq)
			</nowiki>
	view rawmusic_7.py hosted with ❤ by GitHub		view rawmusic_7.py hosted with ❤ by GitHub
	Similarly, prepare the integer sequences for output data as well		Similarly, prepare the integer sequences for output data as well

	unique_y = list(set(y))		<nowiki>unique_y = list(set(y))
	y_note_to_int = dict((note_, number) for number, note_ in enumerate(unique_y))		y_note_to_int = dict((note_, number) for number, note_ in enumerate(unique_y))
	y_seq=np.array([y_note_to_int[i] for i in y])		y_seq=np.array([y_note_to_int[i] for i in y])
Line 406:		Line 412:
	from sklearn.model_selection import train_test_split		from sklearn.model_selection import train_test_split
	x_tr, x_val, y_tr, y_val = train_test_split(x_seq,y_seq,test_size=0.2,random_state=0)		x_tr, x_val, y_tr, y_val = train_test_split(x_seq,y_seq,test_size=0.2,random_state=0)
			</nowiki>

	Model Building		Model Building

Techbot: /* The Workflow of WaveNet: */

2021-08-28T21:17:47Z

The Workflow of WaveNet:

← Older revision		Revision as of 23:17, 28 August 2021
Line 218:		Line 218:
	lstm		lstm

	Pros of LSTM:		= Pros of LSTM =

	Captures the sequential information present in the input sequence		Captures the sequential information present in the input sequence
Line 226:		Line 226:


	Implementation – Automatic Music Generation using Python		= Implementation – Automatic Music Generation using Python =
	The wait is over! Let’s develop an end-to-end model for the automatic generation of music. Fire up your Jupyter notebooks or Colab (or whichever IDE you prefer).		The wait is over! Let’s develop an end-to-end model for the automatic generation of music. Fire up your Jupyter notebooks or Colab (or whichever IDE you prefer).

	Download the Dataset:		= Download the Dataset: =

	I downloaded and combined multiple classical music files of a digital piano from numerous resources. You can download the final dataset from here.		I downloaded and combined multiple classical music files of a digital piano from numerous resources. You can download the final dataset from here.

	Import libraries:		= Import libraries =

	Music 21 is a Python library developed by MIT for understanding music data. MIDI is a standard format for storing music files. MIDI stands for Musical Instrument Digital Interface. MIDI files contain the instructions rather than the actual audio. Hence, it occupies very little memory. That’s why it is usually preferred while transferring files.		Music 21 is a Python library developed by MIT for understanding music data. MIDI is a standard format for storing music files. MIDI stands for Musical Instrument Digital Interface. MIDI files contain the instructions rather than the actual audio. Hence, it occupies very little memory. That’s why it is usually preferred while transferring files.
Line 242:		Line 242:
	Let’s define a function straight away for reading the MIDI files. It returns the array of notes and chords present in the musical file.		Let’s define a function straight away for reading the MIDI files. It returns the array of notes and chords present in the musical file.

	#defining function to read MIDI files		<nowiki>#defining function to read MIDI files
	def read_midi(file):		def read_midi(file):

Line 277:		Line 277:
	return np.array(notes)		return np.array(notes)
	view rawmusic_1.py hosted with ❤ by GitHub		view rawmusic_1.py hosted with ❤ by GitHub
			</nowiki>

Techbot: /* Pros of 1D Convolution */

2021-08-28T21:15:40Z

Pros of 1D Convolution

← Older revision		Revision as of 23:15, 28 August 2021
Line 162:		Line 162:
	Note: The pros and Cons I mentioned here are specific to this problem.		Note: The pros and Cons I mentioned here are specific to this problem.

			= 1D Causal Convolution =

	~~What is~~ 1D Causal Convolution?

	This is defined as convolutions where output at time t is convolved only with elements from time t and earlier in the previous layer.		This is defined as convolutions where output at time t is convolved only with elements from time t and earlier in the previous layer.

Line 171:		Line 168:

	ca		ca
	Pros of Causal 1D convolution:		= Pros of Causal 1D convolution =

	Causal convolution does not take into account the future timesteps which is a criterion for building a Generative model		Causal convolution does not take into account the future timesteps which is a criterion for building a Generative model
Line 181:		Line 178:
	As you can see here, the output is influenced by only 5 inputs. Hence, the Receptive field of the network is 5, which is very low. The receptive field of a network can also be increased by adding kernels of large sizes but keep in mind that the computational complexity increases.		As you can see here, the output is influenced by only 5 inputs. Hence, the Receptive field of the network is 5, which is very low. The receptive field of a network can also be increased by adding kernels of large sizes but keep in mind that the computational complexity increases.

	~~This drives us to the awesome concept of the~~ Dilated 1D Causal Convolution.		= Dilated 1D Causal Convolution =



	~~What is Dilated 1D Causal Convolution?~~

	A Causal 1D convolution layer with the holes or spaces in between the values of a kernel is known as Dilated 1D convolution.		A Causal 1D convolution layer with the holes or spaces in between the values of a kernel is known as Dilated 1D convolution.
Line 195:		Line 188:
	As you can see here, convolving a 3 * 3 kernel over a 7 * 7 input with dilation rate 2 has a reception field of 5 * 5.		As you can see here, convolving a 3 * 3 kernel over a 7 * 7 input with dilation rate 2 has a reception field of 5 * 5.

	Pros of Dilated 1D Causal Convolution:		= Pros of Dilated 1D Causal Convolution =

	The dilated 1D convolution network increases the receptive field by exponentially increasing the dilation rate at every hidden layer:		The dilated 1D convolution network increases the receptive field by exponentially increasing the dilation rate at every hidden layer:

Techbot: /* 1D Convolution */

2021-08-28T21:14:30Z

1D Convolution

← Older revision		Revision as of 23:14, 28 August 2021
Line 144:		Line 144:

	When we set the padding valid, the input and output sequences vary in length. The length of an output is less than an input:		When we set the padding valid, the input and output sequences vary in length. The length of an output is less than an input:
			[[File:Conv-valid.jpg\|thumb\|alt=When we set the padding <strong>valid</strong>, the input and output sequences vary in length. The length of an output is less than an input\|When we set the padding <strong>valid</strong>, the input and output sequences vary in length. The length of an output is less than an input]]

	When we set the padding to same, zeroes are padded on either side of the input sequence to make the length of input and output equal:		When we set the padding to same, zeroes are padded on either side of the input sequence to make the length of input and output equal:
Line 149:		Line 150:
	conv		conv

	Pros of 1D Convolution:		= Pros of 1D Convolution =

	Captures the sequential information present in the input sequence		Captures the sequential information present in the input sequence

Techbot: /* Why and What is a Convolution? */

2021-08-28T21:10:19Z

Why and What is a Convolution?

← Older revision		Revision as of 23:10, 28 August 2021
Line 136:		Line 136:
	convolution		convolution

	~~What is~~ 1D Convolution?		= 1D Convolution =

	The objective of 1D convolution is similar to the Long Short Term Memory model. It is used to solve similar tasks to those of LSTM. In 1D convolution, a kernel or a filter moves along only one direction:		The objective of 1D convolution is similar to the Long Short Term Memory model. It is used to solve similar tasks to those of LSTM. In 1D convolution, a kernel or a filter moves along only one direction:

	convolution		[[File:Calculations-involved-in-a-1D-convolution-operation-300x212.png\|thumb\|alt=Calculations involved in a 1D convolution operation\|Calculations involved in a 1D convolution operation]]

	The output of convolution depends upon the size of the kernel, input shape, type of padding, and stride. Now, I will walk you through different types of padding for understanding the importance of using Dilated Causal 1D Convolution layers.		The output of convolution depends upon the size of the kernel, input shape, type of padding, and stride. Now, I will walk you through different types of padding for understanding the importance of using Dilated Causal 1D Convolution layers.

Techbot: /* Why and What is a Convolution? */

2021-08-28T21:07:48Z

Why and What is a Convolution?

← Older revision		Revision as of 23:07, 28 August 2021
Line 127:		Line 127:
	For example, in the case of image processing, convolving the image with a filter gives us a feature map.		For example, in the case of image processing, convolving the image with a filter gives us a feature map.

	~~music~~ convolving		[[File:Conv output.jpg\|thumb\|alt=In the case of image processing, convolving the image with a filter gives us a feature map\|In the case of image processing, convolving the image with a filter gives us a feature map]]

	Convolution is a mathematical operation that combines 2 functions. In the case of image processing, convolution is a linear combination of certain parts of an image with the kernel.		Convolution is a mathematical operation that combines 2 functions. In the case of image processing, convolution is a linear combination of certain parts of an image with the kernel.
Line 209:		Line 209:

	wavenet		wavenet



	= The Workflow of WaveNet: =		= The Workflow of WaveNet: =

Techbot: /* Output of the WaveNet */

2021-08-28T21:04:11Z

Output of the WaveNet

← Older revision		Revision as of 23:04, 28 August 2021
Line 101:		Line 101:

	[[File:Chunk.png\|thumb\|alt=Input and Output of first 3 chunks\|Input and Output of first 3 chunks]]		[[File:Chunk.png\|thumb\|alt=Input and Output of first 3 chunks\|Input and Output of first 3 chunks]]

	~~Input and Output of first 3 chunks~~

	We can follow a similar procedure for the rest of the chunks.		We can follow a similar procedure for the rest of the chunks.

Techbot: /* Approach 2: Using Long Short Term Memory (LSTM) Model */

2021-08-28T21:03:52Z

Approach 2: Using Long Short Term Memory (LSTM) Model

← Older revision		Revision as of 23:03, 28 August 2021
Line 79:		Line 79:


	Wavenet: The Training Phase		== Wavenet: The Training Phase ==
	This is a Many-to-One problem where the input is a sequence of amplitude values and the output is the subsequent value.		This is a Many-to-One problem where the input is a sequence of amplitude values and the output is the subsequent value.

	Let’s see how we can prepare input and output sequences.		Let’s see how we can prepare input and output sequences.

	Input to the WaveNet:		== Input to the WaveNet ==

	WaveNet takes the chunk of a raw audio wave as an input. Raw audio wave refers to the representation of a wave in the time series domain.		WaveNet takes the chunk of a raw audio wave as an input. Raw audio wave refers to the representation of a wave in the time series domain.
Line 92:		Line 92:
	[[File:Wave.gif\|thumb\|alt=An animated Sound Wave\|An animated Sound Wave]]		[[File:Wave.gif\|thumb\|alt=An animated Sound Wave\|An animated Sound Wave]]

	Output of the WaveNet:		== Output of the WaveNet ==

	Given the sequence of the amplitude values, WaveNet tries to predict the successive amplitude value.		Given the sequence of the amplitude values, WaveNet tries to predict the successive amplitude value.
Line 100:		Line 100:
	The below diagram illustrates the input and output sequences for the model:		The below diagram illustrates the input and output sequences for the model:

	~~chunk~~		[[File:Chunk.png\|thumb\|alt=Input and Output of first 3 chunks\|Input and Output of first 3 chunks]]

	Input and Output of first 3 chunks		Input and Output of first 3 chunks