GPT-J-6B-Skein-4bit-128g

Things I noticed with GPT-J-6B-Skein-4bit when compared to OPT-6B-nerys-v2
	-Outputs special characters even if previous context doesn't have any
	-Original text style isn't always reflected in newly generated text
	-Writes names wrong e.g. Arktinu -> Arkkinu or Semuanya -> Semua
	-Uses wrong pronouns even if this information is in W info
	-Sometimes splits actions such "I go to home. I open the box." rather than writing "I go to home and open the box." even in story mode
	-Does not generate dialogue tags
	-"LayerNormKernelImpl" not implemented for 'Half'
 
Quantization Details

#GTX 1080 8GB
$time python gptj.py ${MODEL_DIR} c4 --nsamples 48 --wbits 4 --groupsize 128 --save ${OUT_DIR}/GPT-J-6B-Skein-4bit-128g/4bit-128g.pt
#real	25m0.999s

Benchmarks
nvidia gtx 1080 8GB, ryzen 5900x, tridenz 4x8gb 3200mhz 16cl

0cc4m KoboldAI
UI2 experimental
Game mode Story
output length 78
context tokens 2048
top_p 1 top_k 0 tail_free 0.95
rep_pen 1.19
rep_pen_range
RNG seed 1
Full Determinism On (despite of this, only few first sentences are identical after restart)

Highest tokens per second:
	model		- layers gpu/disk - tps
OPT-2.7B-Nerys-v2 	- layers 29/3 - 8.44 tokens per second
OPT-6B-nerys-v2 	- layers 8/24 - 1.09 tokens per second
GPT-J-6B-Skein 		- layers 7/21 - 1.18 tokens per second
GPT-J-6B-Skein-4bit - layers 26/2 - 5.06 tokens per second

OPT-6B-nerys-v2-4bit - out of vram and unable to use ram: Expected all tensors to be on the same device, but found at least two, cuda:0 and cpu! devices

Text generation
Generated in +1000 action story with in depth W info that the 2048 context tokens are definitely used up

Prompt: The morning I give the herbs to Kura. "You might find use for these." I say.

 OPT-2.7B-Nerys-v2
	"Thanks, I appreciate it."
	I go back inside, and grab my cloak and helmet.
	When I stand up I notice that Kura has already taken the helm.
	"Hey, I don't need it."
	"No, but I do. You are the leader now and I will follow you."
	"Well, I guess that makes sense."
	"

 OPT-6B-nerys-v2
	"Thank you, I appreciate it."
	I start walking towards the road.
	Kura says "Aren't you going to get revenge?"
	"No, I got nothing against her, she was just trying to scare me."
	"Good idea."
	"Yes, let's move."
	After a short walk I feel the ground shaking beneath my feet.
	I turn around

 GPT-J-6B-Skein
	"Thanks, I'll take care of the rest." He replies.
	"Good luck." I say.
	"Thank you."

	Kura departs from me and I watch him disappear into the horizon. I walk to the edge of the cliff and look out over the desert.
	I think that I could spend the rest of my days here.
	I wonder

 GPT-J-6B-Skein-4bit-128g
	"Thanks, I owe you for saving me. I promise to repay the debt."
	"No need to worry, I know I am not worth enough."
	"How much are you willing to pay?"
	"Nothing, but I wish to help you somehow."
	"That's very kind of you."
	"I shall think of something."

	After talking for



License Apache License 2.0 

Original Model https://huggingface.co/KoboldAI/GPT-J-6B-Skein