Remote Ethereum Node On AWS

[atonughosh]

How do I connect to the remote Ethereum node on AWS? I want to test it.

[drandreaskrueger]

to the remote Ethereum node on AWS?

not sure I understand.

Do these cloud node benchmarking instructions help you perhaps?
https://github.com/drandreaskrueger/chainhammer/blob/master/docs/azure.md

[atonughosh]

Hi,
I hope you're good in this pandemic situation.

Yes, I have deployed the Ethereum nodes on AWS EC2. I could connect to the RPC by modifying the config.py file.
I also launched the test. Used this command from your video on YouTube.
Command Used by Me: CH_TXS=5000 CH_THREADING="threaded2 20" ./run.sh experiment01 geth-clique

Attached is the result.

My gas usage is reaching the limit. Please let me know what is wrong. I am on my private Ethereum network. Ethereum v1.9.
Using AWS t3.2xl, Ubuntu 18.04 for ethereum nodes. 2 nodes. Both mining.

[drandreaskrueger]

Great. At first sight, that looks good. Happy for you, that you made it this far. And all on your own. Well done.

Perhaps you can help me to make the instructions better? Your initial question sounded as if you had been stuck somewhere near the beginning?

Anyways ...:

My gas usage is reaching the limit.

Yes, that's what the fourth diagram was intended for, to be able to detect such situations. Nice.

Let's look into the details. Please click on every link, and have a longer look at each of the files.

Your beginning is: ./run.sh experiment01 geth-clique which is passing on your choice geth-clique here:

chainhammer/run.sh

Line 61 in c17a342

source networks/$2-start.sh

so networks/geth-clique-start.sh is called to start the network. Specifically the docker-compose in this cloned repo:

chainhammer/networks/geth-clique-start.sh

Line 7 in cb1a3cd

FOLDER=networks/repos/drandreaskrueger_geth-dev

So let's look into that repo, which was cloned during the initial setup phase

chainhammer/scripts/install-network-starters.sh

Lines 23 to 29 in 47d4f3a

	echo
	rm -rf drandreaskrueger_geth-dev
	echo javahippie/geth-dev my configured fork drandreaskrueger/geth-dev
	git clone https://github.com/drandreaskrueger/geth-dev.git drandreaskrueger_geth-dev
	cd drandreaskrueger_geth-dev
	echo TODO: Perhaps instead clone from upstream and patch with chainhammer-specific parameters?
	cd ..

Now, when we look into that repo https://github.com/drandreaskrueger/geth-dev.git ...

... we find the genesis.json file with the line "gasLimit": "0x2625A00"

It's been quite a while so I might misremember how Ethereum works, but I think that decimal 40000000 is the value to change. If not, please get back to me.

And:

Keep me in the loop please. May I ask what is the purpose of your trying out chainhammer?
Thanks.

[drandreaskrueger]

Ah, I see. 1 seconds block time. So you have already changed the genesis ? Then look around where you did that choice of 1 second, the gasLimit cannot be far from there too.

However - I have my doubts whether 1 second per block is a good idea.

What are the specs that the geth creators are giving for their current version (of geth clique), in terms of blocktime? Perhaps find their chat, and ask them?

[atonughosh]

Great. At first sight, that looks good. Happy for you, that you made it this far. And all on your own. Well done.

Perhaps you can help me to make the instructions better? Your initial question sounded as if you had been stuck somewhere near the beginning?

Anyways ...:

My gas usage is reaching the limit.

Yes, that's what the fourth diagram was intended for, to be able to detect such situations. Nice.

Let's look into the details. Please click on every link, and have a longer look at each of the files.

Your beginning is: ./run.sh experiment01 geth-clique which is passing on your choice geth-clique here:

chainhammer/run.sh

Line 61 in c17a342

source networks/$2-start.sh

so networks/geth-clique-start.sh is called to start the network. Specifically the docker-compose in this cloned repo:

chainhammer/networks/geth-clique-start.sh

Line 7 in cb1a3cd

FOLDER=networks/repos/drandreaskrueger_geth-dev

So let's look into that repo, which was cloned during the initial setup phase

chainhammer/scripts/install-network-starters.sh

Lines 23 to 29 in 47d4f3a

	echo
	rm -rf drandreaskrueger_geth-dev
	echo javahippie/geth-dev my configured fork drandreaskrueger/geth-dev
	git clone https://github.com/drandreaskrueger/geth-dev.git drandreaskrueger_geth-dev
	cd drandreaskrueger_geth-dev
	echo TODO: Perhaps instead clone from upstream and patch with chainhammer-specific parameters?
	cd ..

Now, when we look into that repo https://github.com/drandreaskrueger/geth-dev.git ...

... we find the genesis.json file with the line "gasLimit": "0x2625A00"

It's been quite a while so I might misremember how Ethereum works, but I think that decimal 40000000 is the value to change. If not, please get back to me.

And:

Keep me in the loop please. May I ask what is the purpose of your trying out chainhammer?
Thanks.

Hi,
Thanks for the awesome proposal of helping you with the instructions.
Yes, initially I was stuck with in beginning.
I'll just have a look into the resources that you shared and try to figure out what's going wrong.
I shall experiment more with it and shall keep you posted.

Attached is my genesis file that I have used to initialize my private network.

I am an MTech student in Computer Science & Engineering from India.
I'm working on my project. I have proposed a system for Cloud Forensics that is Blockchain Based. I need to benchmark my proposed system. I found your tool through Google. I liked it. Also I liked your video on YouTube. That's why I'm using it.

Thanks again for asking me about the instructions. I'm more than happy to help you.

[drandreaskrueger]

I see. Thanks.

Thanks for the explanations why you are using chainhammer. Good to know. It can very probably help you to optimise your throughput. If you want to, please write a chapter or a page about your project, I would like to link to it in other-projects.md#projects-using-chainhammer. No hurry, just get back to me eventually. Thanks a lot.

... Cloud Forensics that is Blockchain Based. I need to benchmark my proposed system.

Ah, cool.
I guess you have your own smart contract for that?

What about this idea:
Identify the probable bottleneck in your system (usually that's stuff related to (either bots running often and in many places concurrently, or) end consumers as they are the largest group in most cases. But even their several smart contract calls are of a very different nature, usually. So: Choose the most frequent one. Let's call that candidate MyContract.bottleneck(args) for now.).

For that, you swap out my "store a simple number" call out of the python send.py code, and instead swap in a call to your MyContract.bottleneck(args) into the mass sending routine.
Have a look here:

chainhammer/hammer/send.py

Lines 189 to 194 in e999a7d

	# CHOOSE which route to choose (web3 / RPC) depending on constant ROUTE
	contract_set = contract_set_via_web3 if ROUTE=="web3" else contract_set_via_RPC

and here:

chainhammer/hammer/contract.sol

Lines 9 to 17 in f8e5d04

	function set(uint x) {
	storedData = x; // uses ~26691 gas
	// try failing transactions:
	// assert ( 1 == 0 ); // uses up all 90000 given gas
	// revert(); // uses 41686 gas
	// throw; // same as revert();
	// require ( 1 == 0 ); // uses 41714 gas
	}

Benchmarking your own call will give you a much more realistic assessment, as it's adapted to your system. But that is step 2. Let's first look into optimising your blockchain setup while still being SmartContract agnostic.

"gasLimit" : "2100000"

ah, so decimal values are allowed too. Good to know.

By now, you have tried increasing that value, right?
You must purge the whole chain probably, and restart with a new chain.

Please tell me when you got rid of your hitting the block limits.

Another thing: You do not have a

"config" : ... "clique": { "period": 5, "epoch": ... }

so you are running it with proof of work - with miners? What is your rationale for that? How can you guarantee enough mining power to not be overwhelmed by malicious miners, taking over your system? Or do you target the Ethereum public chain eventually?

Will your system NEED the proof of work properties (e.g. censorship resistance), or are you using blockchain due to other reasons (e.g. immutability, security, consensus, notarisation, etc). In other words ... have you looked into proof of authority yet?

Also I liked your video on YouTube. That's why I'm using it.

Nice one. I am not particularly proud of that boring and primitive video - but I think it does the trick, of giving a short enough introduction into some of the moving parts of this system.

Yes, initially I was stuck with in beginning.

How could I have made that easier for you?

[atonughosh]

Hello,

I was waiting for your reply :)
I'm learning a lot from you.
Thanks in advance.

If you want to, please write a chapter or a page about your project...

I would love to do that. Once I'm satisfied with my project's functionality and performance. I am surely going to write it.

I guess you have your own smart contract for that?

Yes, I have my own smart contract.

What about this idea:
Please tell me when you got rid of your hitting the block limits.

I was caught up with the project report writing, resuming the work today. Shall let you know all signs of progress.

"config" : ... "clique": { "period": 5, "epoch": ... }

I don't understand this part. Could you please explain that?

are you using blockchain due to other reasons (e.g. immutability, security, consensus, notarisation, etc)

I'm using "Immutability". That is what I need the most. But I'm open to better algorithms as long as I'm getting the Immutability out of my system. But immutability is IMPORTANT.

How could I have made that easier for you?

Your presentation was nice. Calm and composed. Moreover, I followed the directories that you were moving in and the commands that you issued. I related those to my experiment.

I just figured out, I couldn't understand one more thing in the graph generated by Chainhammer. Attached is the TPS graph. I rewatched the video on YouTube too. There you said "the transactions have been smoothed over...." I do not understand the legend of the graph. Why are there 4 types of them? "TPS_1blk", "TPS_3blks", "TPS_5blks" and "TPS_10blks". What are these?

[drandreaskrueger]

I mentioned this, because your genesis.json did not contain the config.clique definitions:

"config" : ... "clique": { "period": 5, "epoch": ... }

I don't understand this part. Could you please explain that?

compare my genesis.json https://github.com/drandreaskrueger/geth-dev/blob/master/miner/genesis.json with yours. Yours is lacking that.

I am a bit disconnected now from Ethereum stuff, it's been a long time ago that I did all this. I am actually not sure WHERE the geth is switched from ProofOfWork algo to the ProofOfAuthority algo "clique".

Here are the commandline switches: (*)
https://github.com/drandreaskrueger/geth-dev/blob/master/miner/Dockerfile#L27
https://github.com/drandreaskrueger/geth-dev/blob/master/node/Dockerfile#L27
https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options

But as the commandline switches used there (*) do not contain anything that could cause that switching of geth from PoW to PoA ... I suspect that it's the genesis.json which makes geth run PoA instead of PoW. But please do your own research, and find it out for sure. Then update me please. Thanks. Actually ... probably it's all here: https://geth.ethereum.org/docs/interface/private-network
. If not, find their chat (gitter? riot? slack?) and just ask them.

---

you said "the transactions have been smoothed over...." I do not understand the legend of the graph.

TPS_1blk is the actual TPS for that single block
TPS_10blks is an averaged TPS, with a window size of 10 blocks.

Why are there 4 types of them?

because I didn't know which ones would turn out as the best ones.

Later I realized it's very much dependent on the situation, so I left them all in.

If your chain is in totally "Zen like" stable tranquillity ... then all of them are ~identical after ~10 blocks. Almost never happens though. Short term fluctuations you'll better see in the blue and yellow.

Does that make sense?

---

I'm using "Immutability".

Good. I suggest you read up on PoW versus PoA - and then probably you want to go for PoA.

[atonughosh]

compare my genesis.json https://github.com/drandreaskrueger/geth-
OK, doing that right away.

I suspect that it's the genesis.json which makes geth run PoA instead of PoW. But please do your > own research, and find it out for sure. Then update me please

Modifying it too once I find it out. Shall let you know once done with it.

Does that make sense?

Absolutely

Thanks a lot.

[atonughosh]

Hello,

Fixed the Gas usage issue.

I think the block time is not super stable. Why is that?

Please provide your valuable feedback on the results for the attached results for 5k and 10k transactions performed with PoW with Ethereum deployed on AWS EC2 t3.2xlarge instance.

Thanks & Regards,
Atonu Ghosh

[drandreaskrueger]

Fixed the Gas usage issue.

Very good. You found the correct spot to tweak it. Where was it?

---

Please provide your valuable feedback ... performed with PoW

I have zero experience with trying chainhammer on PoW. It was still a TODO (see docs/TODO.md).
So what you are trying is actually quite interesting. Please you explore that more. Perhaps you can document it in a long markdown file, similar to these: results/......md <== while trying to understand what exactly is going on, I documented each run. Just chronologically, one above the other. You see what I mean? You could do something similar while exploring ProofOfWork.

---

5k and 10k transactions

Better do more. Or ... many more.
You don't only want to have a handful of blocks.

---

the block time is not super stable. Why is that?

You must read up on the concept of "difficulty".
In PoW, there is actually no "block time" as input parameter. Instead it is a consequence of difficulty and hashpower, and the stochastic process of mining.

[drandreaskrueger]

P.S.:

plus ... not 100% sure if that is the case here (to find out, just download any block, and explore its datastructure) but the timestamp in the blockheader of some of the ethereum/parity/quroum/geth/etc.... versions ... is actually not float but integer.

That could be the reason why you see so many blocktime 1.

Increase the difficulty much, and that will change.

[atonughosh]

Hello,

Very good. You found the correct spot to tweak it. Where was it?

I got it done by setting a larger gas limit in the genesis block.

You see what I mean?

Yes, I do understand. I'm preparing it.
Just to confirm, "CH_TXS=5000 CH_THREADING="threaded2 20" ./run.sh experiment01 geth-clique" issuing this command is in turn giving me results of my PoW implemented?

Better do more. Or ... many more.

Performing more experiments now.

Instead it is a consequence of difficulty and hashpower, and the stochastic process of mining.

Got it and shall find out more on it.

Thanks & Regards,
Atonu Ghosh

[drandreaskrueger]

Just to confirm, "CH_TXS=5000 CH_THREADING="threaded2 20" ./run.sh experiment01 geth-clique" issuing this command is in turn giving me results of my PoW implemented?

To find out, let's do one thing: You (*) reboot (so that you machine is in a defined state) and delete the content of your chainhammer/logs folder. Then you run

CH_TXS=20000 CH_THREADING="threaded2 20" ./run.sh experiment01 geth-clique

and (perhaps additionally copy-paste everything from stdout into another file, and) zip all log files, and attach them to a next comment here.

Then I can have a look.

[drandreaskrueger]

plus

history | tail -n 50

and crop it to anything that you did since the reboot (*).