Coding: Difference between revisions

From EMC23 - Satellite Of Love
Jump to navigation Jump to search
 
(39 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Main_Page]]
[[Main_Page]]
== Studio ==
__NOTOC__
=== Hardware ===
== Getting Started==
* [[Bela]]
* [[Arduino]]
* [[Raspberry Pi]]
* [[RTL-SDR]]
 
=== Learning Platform ===
[//:exercism.io exercism.io]
 
=== Live Coding ===
 
==== DJs of the Future Don't Spin Records—They Write Code ====
 
https://www.wired.com/story/algoraves-live-coding-djs/
 
==== LiveCoding Platforms ====
===== [[Squeek]] =====
===== [[P5live.js]]=====
 
=== Creative Coding ===
* https://www.meetup.com/Creative-Coding-amsterdam/
 
Creative Coding Amsterdam is a community of people interested in all creative aspects of computer programming. It is a place to meet for beginners as well as advanced users, for professionals as well as people coding just for fun.
 
 
* https://www.meetup.com/Creative-Coding-Utrecht/
 
Creative Coding Utrecht (CCU) is a recurring meetup and growing community of coders, developers and people generally interested in the creative possbilities of code & technologies. Please join us in exploring how 21st century skills create new expressions that shape our world.
 
* https://github.com/olilarkin/awesome-musicdsp
 
== Tutorials ==


* [[Setting up your development environment]] for VCVrack,VSTs etc for VSCode in a Windows environment
* [[Setting up your development environment]] for VCVrack,VSTs etc for VSCode in a Windows environment
* Learning Platform [//:exercism.io exercism.io]


=== C++ ===
=== C++ ===
Line 40: Line 10:
==== [[VCVRack]] ====
==== [[VCVRack]] ====
==== [[VST Plugins]] ====
==== [[VST Plugins]] ====
==== Frameworks ====
==== [[C ++ Frameworks]] ====
===== JUCE =====
https://juce.com/
is a partially open-source cross-platform C++ application framework, used for the development of desktop and mobile applications. JUCE is used in particular for its GUI and plug-ins libraries. The aim of JUCE is to allow software to be written such that the same source code will compile and run identically on Windows, Mac OS X and Linux platforms. It supports various development environments and compilers. https://en.wikipedia.org/wiki/JUCE
==== DSP ====
 
===== Juce Examples =====
Digital signal processing (DSP) is the use of digital processing, such as by computers or more specialized digital signal processors, to perform a wide variety of signal processing operations. The signals processed in this manner are a sequence of numbers that represent samples of a continuous variable in a domain such as time, space, or frequency.
 
Digital signal processing and analog signal processing are subfields of signal processing. DSP applications include audio and speech processing, sonar, radar and other sensor array processing, spectral density estimation, statistical signal processing, digital image processing, signal processing for telecommunications, control systems, biomedical engineering, seismology, among others.
 
http://docs.juce.com/master/tutorial_dsp_introduction.html
Discover the realm of digital signal processing and audio buffer manipulation. Learn the basics of the JUCE DSP module and how you can incorporate its classes in your own audio application and plugins.
 
 
http://docs.juce.com/master/tutorial_dsp_convolution.html
Add grit to a synthesiser sound by creating harmonic distortion through waveshaping. Learn the basics of convolution to retrieve the sonic characteristics contained in an impulse response.
 
http://docs.juce.com/master/tutorial_dsp_delay_line.html
Implement a realistic string model by means of physical modelling. Incorporate a delay line to create intricate echo patterns in the stereo sound field.
 
 
http://docs.juce.com/master/tutorial_simd_register_optimisation.html
Take advantage of the processor's parallelism to perform single instruction multiple data calculations. Optimise your audio applications without introducing concurrency.
 
http://docs.juce.com/master/tutorial_simple_fft.html
Learn how to display incoming audio data as a spectrogram by using the FFT class of the DSP module. Understand the benefits of using a Fast Fourier Transform.
 
http://docs.juce.com/master/tutorial_spectrum_analyser.html
Learn how to display incoming audio data as a spectrum analyser by using the FFT class of the DSP module. Understand the benefits of using a windowing function.
* VULT


=== Javascript ===
=== Javascript ===
Line 79: Line 19:
=== Python ===
=== Python ===
==== [[Python|Basics]] ====
==== [[Python|Basics]] ====
==== [[Audio Manipulation]] ====


==== [[Deep Learning]] ====
==== [[Deep Learning]] ====
Machine Learning is a subset of A.I


===== Definition =====
=== Max/Pd ===
In traditional problem-solving with software, a person analyzes a problem and engineers a solution in code to solve that problem.
* [[Build a synth in Pure Data]]
In machine learning the problem solver abstracts away part of their solution as a flexible component called a model, and uses a special program called a model training algorithm to adjust that model to real-world data. The result is a trained model which can be used to predict outcomes that are not part of the data set used to train it.
* [[Build a synth in Max For Live]]


===== Applications =====
=== C Sound ===
* Speech Recognition
=== SuperCollider ===
* Voice Based emotion classification
* Noise recognition
* Musical Genre Instrument Mood Classificatiob
* Music Tagging
* Music Generation


====== Supervised Learning ======
== Studio ==
In supervised learning, every training sample from the dataset has a corresponding label or output value associated with it. As a result, the algorithm learns to predict labels or output values.
=== Hardware ===
* [[Bela]]
* [[Arduino]]
* [[Raspberry Pi]]
* [[RTL-SDR]]


====== Unsupervised Learning ======
=== [[Live Coding]] ===
In unsupervised learning, there are no labels for the training data. A machine learning algorithm tries to learn the underlying patterns or distributions that govern the data.


Unsupervised learning involves using data that doesn't have a label. One common task is called '''clustering'''. Clustering helps to determine if there are any naturally occurring groupings in the data.
=== Creative Coding ===
 
* https://www.meetup.com/Creative-Coding-amsterdam/
===== Reinforcement Learning =====
* https://www.meetup.com/Creative-Coding-Utrecht/
In reinforcement learning, the algorithm figures out which actions to take in a situation to maximize a reward (in the form of a number) on the way to reaching a specific goal.
 
An agent is a piece of software you are training that makes decisions in an environment to reach a goal.
 
* An algorithm is a set of instructions that tells a computer what to do. ML is special because it enables computers to learn without being explicitly programmed to do so.
* The training algorithm defines your model’s learning objective, which is to maximize total cumulative reward. Different algorithms have different strategies for going about this.
# * A soft actor critic (SAC) embraces exploration and is data-efficient, but can lack stability.
# 
# * A proximal policy optimization (PPO) is stable but data-hungry.
* An action space is the set of all valid actions, or choices, available to an agent as it interacts with an environment.
# * Discrete action space represents all of an agent's possible actions for each state in a finite set of steering angle and throttle value combinations.
#
# Continuous action space allows the agent to select an action from a range of values that you define for each state.
 
* Hyperparameters are variables that control the performance of your agent during training. There is a variety of different categories with which to experiment. Change the values to increase or decrease the influence of different parts of your model.
# * For example, the learning rate is a hyperparameter that controls how many new experiences are counted in learning at each step. A higher learning rate results in faster training but may reduce the model’s quality.
 
* The reward function's purpose is to encourage the agent to reach its goal. Figuring out how to reward which actions is one of your most important jobs.
 
The more an agent learns about its environment, the more confident it becomes about the actions it chooses.
 
If an agent doesn't explore enough, it often sticks to information its already learned even if this knowledge doesn't help the agent achieve its goal.
 
The agent can use information from previous experiences to help it make future decisions that enable it to reach its goal.
 
=== Machine Learning  steps for music generation ===
ML is a generic problem solver. A model can solve many problems inclusding ones not dicovered until the model is in action.
The Model is created from data, through an iterative process the model is 'fitted' to the edata. With the final model inferences are made.
 
* Define the Problem
* Build the Dataset
* Train the M odel
* Evaluate the Model
* Use the Model
 
A task is supervised if you are using '''labeled''' data. We use the term labeled to refer to data that already contains the solutions, called labels.
In supervised learning, there are two main identifiers you will see in machine learning:
 
    A categorical label has a discrete set of possible values. Furthermore, when you work with categorical labels, you often carry out classification tasks*, which are part of the supervised learning family.
 
    A continuous (regression) label does not have a discrete set of possible values, which often means you are working with numerical data.
 
# Data Collection
Does the data you've collected match the machine learning task and problem you have defined?
# Data Inspection
 
* Outliers
* Missing or incomplete values
* Data that needs to be transformed or preprocessed so it's in the correct format to be used by your model
 
# Summary Statistics
check that your data is in line with the underlying assumptions
With many statistical tools, you can calculate things like the mean, inner-quartile range (IQR), and standard deviation. These tools can give you insight into the scope, scale, and shape of the dataset.
# Data Visualisation
You can use data visualization to see outliers and trends in your dat
 
Splitting your dataset gives you two sets of data:
    Training dataset: The data on which the model will be trained. Most of your data will be here. Many developers estimate about 80%.
    Test dataset: The data withheld from the model during training, which is used to test how well your model will generalize to new data.
 
The model training algorithm iteratively updates a model's parameters to minimize some loss function.
 
==== Model parameters ====
: Model parameters are settings or configurations the training algorithm can update to change how the model behaves, such as weights and biases. Weights, which are values that change as the model learns, are more specific to neural networks.
 
==== Loss function ====
A loss function is used to codify the model’s distance from this goal. 
 
==== Model Types ====
* Linear models
* Tree-based models
* Deep learning models
    * FFNN: The most straightforward way of structuring a neural network, the Feed Forward Neural Network (FFNN) structures neurons in a series of layers, with each neuron in a layer containing weights to all neurons in the previous layer.
    *    CNN: Convolutional Neural Networks (CNN) represent nested filters over grid-organized data. They are by far the most commonly used type of model when processing images.
    *    RNN/LSTM: Recurrent Neural Networks (RNN) and the related Long Short-Term Memory (LSTM) model types are structured to effectively represent for loops in traditional computing, collecting state while iterating over some object. They can be used for processing sequences of data.
    *    Transformer: A more modern replacement for RNN/LSTMs, the transformer architecture enables training over larger datasets involving sequences of data.
 
==== Model Evaluation ====
Log loss seeks to calculate how uncertain your model is about the predictions it is generating.
 
Model Accuracy is the fraction of predictions a model gets right.
 
1. R Square/Adjusted R Square
2. Mean Square Error(MSE)/Root Mean Square Error(RMSE)
3. Mean Absolute Error(MAE)
Things to think about
 
There are many different tools that can be used to evaluate a linear regression model. Here are a few examples:
 
    Mean absolute error (MAE): This is measured by taking the average of the absolute difference between the actual values and the predictions. Ideally, this difference is minimal.
 
    Root mean square error (RMSE): This is similar MAE, but takes a slightly modified approach so values with large error receive a higher penalty. RMSE takes the square root of the average squared difference between the prediction and the actual value.
 
    Coefficient of determination or R-squared (R^2): This measures how well-observed outcomes are actually predicted by the model, based on the proportion of total variation of outcomes.
 
 
 
Accuracy    False positive rate Precision
 
Confusion matrix False negative rate Recall
 
F1 ScoreLog LossROC curve
 
Negative predictive value Specificity
 
==== Model Inference ====
    * Generating predictions.
    * Finding patterns in your data.
    * Using a trained model.
    * Testing your model on data it has not seen before.
 
=== Generative AI Models Used for Music Composition ===
Generative adversarial networks (GANs), general autoregressive models, and transformer-based models.
 
=== Autoregressive models ===
Autoregressive convolutional neural networks (AR-CNNs) are used to study systems that evolve over time and assume that the likelihood of some data depends only on what has happened in the past. It’s a useful way of looking at many systems, from weather prediction to stock prediction.
 
When a note is either added or removed from your input track during inference, we call it an edit event.
 
To train the AR-CNN model to predict when notes need to be added or removed from your input track (edit event), the model iteratively updates the input track to sounds more like the training dataset. During training, the model is also challenged to detect differences between an original piano roll and a newly modified piano roll.
 
==Generative adversarial networks (GANs)==
== Generative adversarial networks (GANs), are a machine learning model format that involves pitting two networks against each other to generate new content. The training algorithm swaps back and forth between training
* a generator network (responsible for producing new data) and
* a discriminator network (responsible for measuring how closely the generator network’s data represents the training dataset).
 
The generator and the discriminator are trained in alternating cycles. The generator learns to produce more and more realistic data while the discriminator iteratively gets better at learning to differentiate real data from the newly created data.
 
    Generator: A neural network that learns to create new data resembling the source data on which it was trained.
    Discriminator: A neural network trained to differentiate between real and synthetic data.
    Generator loss: Measures how far the output data deviates from the real data present in the training dataset.
    Discriminator loss: Evaluates how well the discriminator differentiates between real and fake data.
 
 
== Transformer-based models ==
Transformer-based models are most often used to study data with some sequential structure (such as the sequence of words in a sentence). Transformer-based methods are now a common modern tool for modeling natural language.
 
== Architectural Patterns ==
which could be applied to them)
 
• convolutional,
 
• conditioning,
 
• adversarial.
 
 
From this basic building block, we will describe in the following sections the main types of deep learning architectures used for music generation (as well as for other purposes):
 
• feedforward,
 
• autoencoder,
 
• restricted Boltzmann machine (RBM),
 
• recurrent (RNN).
 
== Python ==
There are several essential tools in the [[Python]] kit for Deep Learning
[[Librosa]] is used to analyse and manipulate audio
 
[[Tensorflow]] is used to train models
 
[[Keras]] High Level Library for Tensorflow
 
[[SkLearn]] statistical library with examples and tutorials
 
 
== Generating Sound with Neural Networks ==
=== Defining the sound generation task ===
=== Classification of sound generation systems ===
==== WaveNet ====
==== Jukebox ====
=== Types of generated sounds ===
=== Sound representations ===
=== Generation from raw audio ===
=== Challenges of raw audio generation ===
 
long-range dependencies
* Pitch
* Melody
* Structure
* Timbre
* Rhythm
* Harmony
 
=== Generation from spectrograms ===
More compact than numeric representation of raw audio
 
Short terms Fourier transform -> Model model ->  spectrogram -> Inverse Short terms Fourier transform ->  audio
 
=== Advantages of generation from spectrograms ===
* Temporal axis of spectrogram is more compact than that of waveform
* Capture longer time dependencies
* Computationally lighter than raw audio
 
=== Challenges of generation from spectrograms ===
* Audio Fidelity
* Phase reconstruction
 
We cannot generate sound with Mel Frequency Cepstral Coefficients or MFCCs
 
=== Deep Learning architectures for sound generation ===
* Gan
* Autoencoder
Encoder compresses original data down to bottleneck or latent space, the decoder decompresses data back to original domain (dimensions) using back propagation, using minimise reconmstuction error
E(x,x1) . The difference between the original data and the reconstructed data and Root mean square error function to reconstruct but regularization to not overfit.
 
from tensorflow.keras import Model
from tensorflow.keras.layers import Input, Conv2D, ReLU, BatchNormalization, \
    Flatten, Dense
from tensorflow.keras import backend as K
 
[[File:AE.png|thumb]]
 
* Variational Autoencoder (VAE)
* VQ-VAE
 
=== Inputs for generation ===
* Conditioning
Uses conditions to create output eg create vocal like x in a style of y
 
* Autonomous
Free wild generation
 
* Continuation
Start a sequence and have model continue
 
== Tutorials ==
=== The Sound of AI ===
* [[Deep Learning (for Audio) with Python]] - 19 videos
 
* [[Audio Signal Processing for Machine Learning]] - 23 videos
* [[PyTorch for Audio + Music Processing]] - 10 videos
* [[Generating Sound with Neural Networks]] - 14 videos
 
[https://www.youtube.com/c/ValerioVelardoTheSoundofAI Valerio Velardo - The Sound of AI]
 
== Links ==
Audio Handling Basics: Process Audio Files In Command-Line or Python
 
https://benhayes.net/projects/nws/#audio-examples<br />
 
Do Androids Dream of Electric Beats?<br />
https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53<br />
Intro to Audio Analysis: Recognizing Sounds Using Machine Learning<br />
https://magenta.tensorflow.org/music-vae<br />
https://musicalmetacreation.org/mume2018/proceedings/Sturm.pdf<br />
https://ccrma.stanford.edu/~blackrse/algorithm.html<br />
https://magenta.tensorflow.org/music-vae<br />
https://mirg.city.ac.uk/codeapps/the-magnatagatune-dataset<br />
https://docs.microsoft.com/en-us/cognitive-toolkit/<br />
https://scikit-learn.org/stable/<br />
https://pythonrepo.com/repo/nerdyrodent-VQGAN-CLIP-python-deep-learning<br />
 
<evlplayer id="player1" w="480" h="360" service="youtube" defaultid="MwtVkPKx3RA" />
 
== Terminology ==
'''Deep learning''' is a class of machine learning algorithms that[12](pp199–200) uses multiple layers to progressively extract higher-level features from the raw input.
 
'''Learning''' can be '''supervised''', '''semi-supervised''' or '''unsupervised'''
 
'''Impute is a common term referring to different statistical tools which can be used to calculate missing values from your dataset.
Outliers are data points that are significantly different from others in the same sample.
 
'''FFNN''': The most straightforward way of structuring a neural network, the Feed Forward Neural Network (FFNN) structures neurons in a series of layers, with each neuron in a layer containing weights to all neurons in the previous layer.<br />
'''CNN''': Convolutional Neural Networks (CNN) represent nested filters over grid-organized data. They are by far the most commonly used type of model when processing images.<br />
'''RNN/LSTM''': Recurrent Neural Networks (RNN) and the related Long Short-Term Memory (LSTM) model types are structured to effectively represent for loops in traditional computing, collecting state while iterating over some object. They can be used for processing sequences of data.<br />
'''Transformer''': A more modern replacement for RNN/LSTMs, the transformer architecture enables training over larger datasets involving sequences of data.<br />
 
'''Bag of words''': A technique used to extract features from the text. It counts how many times a word appears in a document (corpus), and then transforms that information into a dataset.
 
'''A categorical label''' has a discrete set of possible values, such as "is a cat" and "is not a cat."
 
'''Clustering'''. Unsupervised learning task that helps to determine if there are any naturally occurring groupings in the data.
 
'''CNN''': Convolutional Neural Networks (CNN) represent nested filters over grid-organized data. They are by far the most commonly used type of model when processing images.
 
A continuous (regression) label does not have a discrete set of possible values, which means possibly an unlimited number of possibilities.
 
'''Data vectorization''': A process that converts non-numeric data into a numerical format so that it can be used by a machine learning model.
 
'''Discrete''': A term taken from statistics referring to an outcome taking on only a finite number of values (such as days of the week).
 
'''FFNN''': The most straightforward way of structuring a neural network, the Feed Forward Neural Network (FFNN) structures neurons in a series of layers, with each neuron in a layer containing weights to all neurons in the previous layer.
 
'''Hyperparameters''' are settings on the model which are not changed during training but can affect how quickly or how reliably the model trains, such as the number of clusters the model should identify.
 
'''Log loss''' is used to calculate how uncertain your model is about the predictions it is generating.
 
'''Hyperplane''': A mathematical term for a surface that contains more than two planes.
 
'''Impute''' is a common term referring to different statistical tools which can be used to calculate missing values from your dataset.
 
'''label''' refers to data that already contains the solution.
 
'''loss function''' is used to codify the model’s distance from this goal
 
'''Machine learning''', or ML, is a modern software development technique that enables computers to solve problems by using examples of real-world data.
 
'''Model''' accuracy is the fraction of predictions a model gets right. Discrete: A term taken from statistics referring to an outcome taking on only a finite number of values (such as days of the week). Continuous: Floating-point values with an infinite range of possible values. The opposite of categorical or discrete values, which take on a limited number of possible values.
 
'''Model inference''' is when the trained model is used to generate predictions.
 
model is an extremely generic program, made specific by the data used to train it.
 
'''Model parameters''' are settings or configurations the training algorithm can update to change how the model behaves.
 
'''Model training algorithms''' work through an interactive process where the current model iteration is analyzed to determine what changes can be made to get closer to the goal. Those changes are made and the iteration continues until the model is evaluated to meet the goals.
 
'''Neural networks''': a collection of very simple models connected together. These simple models are called neurons. The connections between these models are trainable model parameters called weights.
 
'''Outliers''' are data points that are significantly different from others in the same sample.
 
'''Plane''': A mathematical term for a flat surface (like a piece of paper) on which two points can be joined by a straight line.
 
'''Regression''': A common task in supervised machine learning.
 
In reinforcement learning, the algorithm figures out which actions to take in a situation to maximize a reward (in the form of a number) on the way to reaching a specific goal.
 
'''RNN/LSTM''': Recurrent Neural Networks (RNN) and the related Long Short-Term Memory (LSTM) model types are structured to effectively represent for loops in traditional computing, collecting state while iterating over some object. They can be used for processing sequences of data.
 
'''Silhouette coefficient''': A score from -1 to 1 describing the clusters found during modeling. A score near zero indicates overlapping clusters, and scores less than zero indicate data points assigned to incorrect clusters. A
 
'''Stop words''': A list of words removed by natural language processing tools when building your dataset. There is no single universal list of stop words used by all-natural language processing tools.
 
In '''supervised learning''', every training sample from the dataset has a corresponding label or output value associated with it. As a result, the algorithm learns to predict labels or output values.
 
'''Test dataset''': The data withheld from the model during training, which is used to test how well your model will generalize to new data.
 
'''Training dataset''': The data on which the model will be trained. Most of your data will be here.
 
'''Transformer''': A more modern replacement for RNN/LSTMs, the transformer architecture enables training over larger datasets involving sequences of data.
 
In '''unlabeled data''', you don't need to provide the model with any kind of label or solution while the model is being trained.
 
In '''unsupervised learning''', there are no labels for the training data. A machine learning algorithm tries to learn the underlying patterns or distributions that govern the data.
 
  Machine learning is synthesizing death metal. It might make your death metal radio DJ nervous – but it could also mean music software works with timbre and time in new ways. That news – plus some comical abuse of neural networks for writing genre-specific lyrics in genres like country – next.
 
Peter Kirn http://cdm.link/2019/04/now-ai-takes-on-writing-death-metal-country-music-hits-more/

Latest revision as of 22:34, 10 March 2022

Main_Page

Getting Started[edit]

C++[edit]

Basics[edit]

VCVRack[edit]

VST Plugins[edit]

C ++ Frameworks[edit]

Javascript[edit]

Basics[edit]

Node.js[edit]

Web Audio[edit]

Python[edit]

Basics[edit]

Audio Manipulation[edit]

Deep Learning[edit]

Max/Pd[edit]

C Sound[edit]

SuperCollider[edit]

Studio[edit]

Hardware[edit]

Live Coding[edit]

Creative Coding[edit]

Retrieved from "https://wiki.emc23.com/?title=Coding&oldid=1185"