The temperature parameter is used to control the randomness of the output. The higher the temperature the more random the output will be.
In the above section the LLM, the predicted next word does not have to be the most probable word. For example if we want the LLM to be more creative we can tell it to not always choose the most probable word but instead choose another one to create something different. This is what the temparature parameters controls. So for question answering involving facts we would want the LLM to be always choose the most probable word but for creative writing we would want it to be more creative and thus have a highter temperature value.
This is a hyperparameter that uses a technique called nucleus sampling. This works as follows, we start by sorting the words in our very limited vocabulary:
"apple" : 0.5
"banana" : 0.2
"strawberry": 0.15
"watermelon": 0.1
"mango" : 0.05
Next, we add these probablities until we reach, or exceed the specified top_p
value. For example if top_p = 0.8:
"apple": 0.5 + "banana": 0.2 + "strawberry": 0.15 = 0.85
This means our neucleus will then be:
["apple", "banana", "strawberry"]
And then a random sample is choses from that neucleus. So without this the most probable word would always be chosen but with this we can get some variation in the output.
This is a hyperparameter that uses a technique called top-k sampling. Lets say we have the following vocabulary:
"apple" : 0.4
"banana" : 0.25
"cherry" : 0.15
"date" : 0.1
"elderberry": 0.05
"fig" : 0.02
"grape" : 0.015
"honeydew" : 0.01
"kiwi" : 0.005
"lemon" : 0.005
And say we set top_k = 4. This means that we will only consider the 4 top most
probable words for the next selection and then sample from them.
Unlike Nucleus Sampling (top_p), which considers a variable number of tokens
based on a probability threshold, top_k sampling always considers a fixed
number of tokens (the top k most probable ones).
The advantage of top_k is that it's straightforward and computationally
efficient. However, it may not capture as much diversity as top_p if the top
few tokens have significantly higher probabilities than the rest.
This is introduced in the paper: https://arxiv.org/pdf/2202.00666.pdf. Example, we have the following vocabulary:
"sunny"
"cloudy"
"rainy"
"windy"
In llama.cpp we construct a
float p = 0.8;
size_t min_keep = 4;
struct llama_sampler* sampler = llama_sampler_init_top_p(p, min_keep);This is introduced in the paper: https://openreview.net/pdf?id=W1G1JZEIy5_ and
is also related to sampling of generated tokens, so in the same "category" as
top_p and top_k I think. It is about controlling perplexity, which is a
measure of how well a language model predicts a sample of text (lower is better).
It sounds like MIROSTAT also uses top_k sampling but unlike traditional top-k
sampling, MIROSTAT adaptively adjusts the value of k to control the perplexity.
So there is still a top_k sampling performed but the value of k is not fixed
but instead is adjusted to control the perplexity.
This is a hyperparameter that penalizes the probability of tokens that have already been generated. This is useful for text generation tasks where we want to avoid repetition.
The probability of each token is modified like this:
modified probability = original probabilty^repetition_penalty
Here the original probablity is the probability of the token given to the token by the LLM. The repetition penalty is a value between 1 and infinity. A value of 1 will not do anything as we in this case are just raising the probability to the power of 1 which is just the original probability.
Now, a value greater than one will actually make the probability, a value between 0-1, smaller which might seem counter intuitive but it makes sense if we think about it:
0.20^1.2 = 0.16
0.20^-1.2 = 0.25
-
A Repetition Penalty greater than 1 discourages repetition by reducing the probabilities of already-appeared tokens.
-
A Repetition Penalty less than 1 encourages repetition by increasing the probabilities of already-appeared tokens.
This is a hyperparameter that is similar to the repetition penalty but instead controls the likelihood of generating tokens that are inherently frequent in the language model's training data, whereas repetition penalty is about the tokens in the generated text.
The following is applied to each token in the vocabulary:
modified probability = original probabilty^frequency_penalty
Frequency Penalty is applied universally to all tokens, while Repetition Penalty is applied only to tokens that have already appeared in the generated text.
Frequency Penalty is useful when you want to encourage or discourage the use of common words. Repetition Penalty is useful when you want to avoid repetitive text in longer sequences.
This hyperparameter influences the models output and controls the presence or absence of new token in the generated text.
If this value is 0 then there is no effect at all for this hyperparameter. If this value is > 0, the model is discouraged from introducing new tokens that are not strongly supported by the context (the model is more conservative). If this value is < 0 the model is encouraged to introduce new tokens, even if they are not strongly supported by the context (the model is more creative).
When the model is generating text, it computes a probability distribution over the vocabulary for the next token to be generated, given the current context (i.e., the tokens that have been generated so far and the initial prompt).
The PresencePenalty value is used to adjust these probabilities before the next token is sampled. Specifically, it modifies the "logits" (unnormalized log probabilities) associated with each token in the vocabulary. The adjustment can either increase or decrease the likelihood of each token being selected as the next token in the sequence, depending on the value of PresencePenalty
Somewhat simplified process:
-
The model computes a probability distribution over the vocabulary for each token in the vocabulary based on the current context.
-
The PresencePenalty value is used to adjust these probabilities by adding or subtracting the value from the logit (unnormalized log probabilities) associated with each token.
-
Adjusted/Rescale the logits so they sum to 1 after possibly adding/subtracting the PresencePenalty value.
-
Sample the next token from the adjusted probability distribution.
-
Add the token to the current context and repeat the process.
p("the") = 0.6
p("a") = 0.2
p("it") = 0.15
p("house") = 0.02
p("banana") = 0.0001
... and so on for thousands of other words.Find the max probability, the token with the highest probability, so this would be 'the' with a probability of 0.6.
Next we calculate a cutoff threshold which is what the min_p parameter is used for, which is a user defined value between 0 and 1.
threshold = p_max * min_pIf we choose min_p = 0.1, then threshold = 0.6 * 0.1 = 0.06.
Go through the list of all possible probabilites and discard any of the probablities that are less than the threshold.
p("the") = 0.6 (Keep, it's >= 0.06)
p("a") = 0.2 (Keep, it's >= 0.06)
p("it") = 0.15 (Keep, it's >= 0.06)
p("house") = 0.02 (Discard, it's < 0.06)
p("banana") = 0.0001 (Discard, it's < 0.06)So the float p that this sampler accepts, this is a value between 0 and 1 which we use to multiply by the maxium probablity in the set of logits. This gives us a threshold, and we subtract this from the probabilities. Where the output of this subtraction is zero, or less than zero the step function will produce 0 for the zero or negative values and 1 for the positive which are those that are to be kept
const float large_neg_val = -1e9f;
struct ggml_tensor * punishment = ggml_scale_bias(ctx, mask, -large_neg_val, large_neg_val);
ggml_set_name(punishment, "min_p_punishment");The function ggml_scale_bias(ctx, a, s, b) calculates the formula:
output = (a * s) + b
a is the input tensor.
s is a float scale factor.
b is a float bias to be added.
Lets plug in the values:
a = mask (our tensor of 1.0s and 0.0s)
s = -large_neg_val = -(-1e9f) = 1e9f
b = large_neg_val = -1e9fSo the formula becomes:
punishment = (mask * 1e9f) + (-1e9f)Now, let's see what happens for the two different values inside the mask tensor:
- When we want to KEEP a token:
The mask value for this token is 1.0f.
The calculation is: (1.0f * 1e9f) + (-1e9f)
This simplifies to: 1e9f - 1e9f = 0.0f
Result:
The punishment is 0.0f. When we add this to the original logit, the logit is
unchanged. This is exactly what we want for tokens we keep.
- When we want to DISCARD a token:
The mask value for this token is 0.0f.
The calculation is: (0.0f * 1e9f) + (-1e9f)
This simplifies to: 0.0f - 1e9f = -1e9f
Result:
The punishment is -1e9f. When we add this to the original logit, it gets driven
to a very large negative value. This is exactly what we want for tokens we discard.
We could also reverse the negative value:
a = mask (our tensor of 1.0s and 0.0s)
s = 1e9f
b = -1e9f
So we are first scaling the mask which are values that are either 0 or 1. So only values that are to be kept will:
punishment = (mask * 1e9f) + (-1e9f)
0 * 1e9f + (-1e9f) = -1e9f (bascially -infinity, we discard this token)
1 * 1e9f + (-1e9f) = 0 (nothing happend to tokens we keep)