This paper discusses the notion of sentience in machines, and proposes an approach to analyze it objectively.
It draws insight from studies in Affective Neuroscience which map Functional neuroimaging data on a subject’s brain activity, to their emotional states. It then outlines a procedure to obtain useful information about possible sentience in a given machine/AI model.
It is hoped that this research inspires more work aimed towards structuring an objective test of sentience in machines.
It’s a plastic raft- It’s made of a number of buoyant plastic cuboids strung together somehow.
The raft bobs gently atop the water.
The night’s air is entrancingly tranquil.
The raft bobs gently atop the water.
I’m listening to Billie Eilish’s “Ocean Eyes”.
I’ve got these Motorola Bluetooth headphones I bought online over the winter with the appreciable student-internship money that came in over the holidays.
December was chill. Very chill. I think I discovered Imogen Heap in December. Through Spotify.
It was a very stimulating experience- Having the atmospheric electrification of her robotic music reverberate through the dimly-lit room while I sat at my computer, doing some data-processing internship work against the backdrop of downtown San Francisco’s shimmering night-time skyline.
I think I bought the headphones from some site an Indian classmate told me about. I had never heard of the site before. They had some pretty solid deals.
I like the headphones. Rather than go over my head from ear to ear, they go around the back of my head. I find that really appealing, because other headphones leave this like cuboidal crater across the top of my head whenever I take them off. Because they compress the hair.
The Motorola headphones are cool.
The raft bobs gently atop the water.
I’m listening to Billie Eilish’s “Ocean Eyes”.
At this point in time, I don’t know Billie Eilish by name. She’s one of the artists whose cool songs Spotify recently recommended for me. Or put together in a playlist. Something like that.
“Ocean Eyes” by Billie Eilish.
Along with “Slip” by Elliot Moss.
Haha. I also really like that one. I really like it.
At this point in time, I’m aware of these artists more by the enigmatic communal identity given them by their general genre, than by their names.
The raft bobs gently atop the water.
I’m falling in love.
I don’t know it yet though. I don’t know it yet.
I was at this seminar earlier. At the Nervana HQ. Nervana is this Deep Learning library I recently learnt about. They held a seminar to publicize their library and familiarize people with its workings.
I don’t know anything about Deep Learning. I just know I’m very interested in it.
I got to the seminar late. Everyone was seated and listening intently to what some Indian guy was saying. That was when I walked into the room. I strolled in and found a seat for myself at the back.
“I’m going to make you late for your seminar.”
It was something of a concerned whisper.
An anxious wisp of a voice that floated its way out of an entanglement of smudged lipstick and sensual gasps and intertwined limbs and an unhooked brassiere.
“Don’t worry, the seminar isn’t all that important.”
The intertwined limbs kept at whatever it was they were doing, relatively uninterrupted.
I’m falling in love.
I don’t know it yet though.
I don’t know it yet.
The people at the seminar probably saw me stride regally across the room, looking very esteemed and cool and untroubled and confident.
What they probably didn’t know was that my otherworldly chill and dreamy tranquility at the time, came from engagement in a completely different kind of activity. I new next to nothing about the technical topics being discussed in the room.
General view from the floating plastic raft
The raft bobs gently atop the water.
“Ocean Eyes” by Billie Eilish keeps playing.
I don’t know how much time passes before I get off the raft. Maybe an hour. Maybe more. I get off, and on to the small pier leading back to the road.
I keep walking along the bay.
At some point I come across this guy. He’s standing right in front of the railings by the sidewalk. He’s facing the water.
I think he’s fishing.
Yes. Yes, he’s fishing.
He’s got this fishing rod in his hands.
Somehow, we begin to converse.
He is from the Dominican Republic.
I think.
Some Latin American country.
We talk about general things for a bit.
He works construction in the US.
At some point he tells me about his girlfriend:
He used to have a girlfriend. Back in his country of origin.
They lost touch when he moved to the US for work.
Her mother never liked him.
Hated him actually. For some reason.
Hated him so much that she prevented him from ever knowing he had a daughter.
Something like that.
His girlfriend was pregnant at the time he moved to the US.
But he never got to know.
He just recently found out.
He reunited with his girlfriend the year before. They spent the holidays together somewhere in Latin America.
He shows me pictures and videos.
His daughter is like six years old.
Or seven.
Or eight even, I don’t know.
I’m shocked he could have been deprived of news of her existence for that long.
He smiles as he scrolls through the pictures, his face gleaming with immense fondness for the both of them. He says he’s planning towards having them join him in the US.
We keep talking.
At some point he asks for my story.
“What’s your own story?” He asks, curious.
I tell him I’m studying in the US on a scholarship.
He seems very wowed. Very very wowed. Immensely impressed.
We keep talking.
At some point I try my hand at his fishing rod.
I’ve never done anything of the sort before.
He gives me some instructions: Do this, do this, after that do this and do this…
He lists like seven different steps.
I find myself repeating the steps after him, but I know only like the first one actually made it into my head.
I swing the rod to send the bait and hook thing into the water.
There is an unnerving “CRACK!” somewhere along the equipment.
I think I just broke something.
I begin to apologize profusely.
“Oh my God I’m so sorry, that was so clumsy of me I’m so sorry I hope it’s not something so serious”
He says there’s no problem. That I shouldn’t worry about it.
His reaction gives me some relief.
We keep talking.
At some point he asks where I was headed.
I point in the general direction to the right.
He says okay.
He says okay, but that I shouldn’t go too much further down that path.
He says there’s a very dangerous area there- Third street.
Says the place is very unsafe, especially at night. I should avoid it at all costs.
I thank him for his concern.
We keep talking.
A number of weeks later, I’ll find myself skateboarding down the infamous Third Street.
At night.
I never planned it- I was just skateboarding along the bay, and then somehow I found myself there.
As I cruise down the sidewalk on my skateboard, I’ll take my time to observe the people I come across. They’re all black. They look like the sorts of people you’ll come across walking through the Tenderloin district- the sorts of people the Terderloin is kinda known for, but they don’t look particularly menacing.
Some of them look vaguely sketchy- not exactly the type you’d think of walking up to and striking up a conversation with.
I don’t know- Maybe they look particularly unnerving precisely because of the pronounced warning of the Latin American fishing guy.
I don’t know.
Thoughts of getting some fast food at a road-side restaurant will cross my mind.
I’ll be concerned about the time. It’ll be pretty late.
Usually I’m out at midnight in San Francisco and don’t feel uneasy at all.
But this is Third Street.
I’ve only ever heard one thing about this place. And that, is that I should not do exactly what I’m doing right now.
I won’t buy food.
I’ll skateboard down Third Street some more.
I’ll come across a MUNI stop.
Ah, a train goes back towards the Ferry building from here.
I’ll wait at the stop- skateboard in hand. Wondering what my fate will be, surrounded by all of the suspicious Third street people.
The train will take forever to come.
Image: View of Downtown San Francisco’s shimmering night-time skyline from one of the dorm room windows.
Regularization seems to be it for this model+dataset.
Like, nothing else seems to delay overfitting as effectively at it does.
God I’ve been trying stuff.
Broader layers just over fit like—— that. Pam pam pam. Like fifty epochs and your validation loss is beginning to rise.
Network depth seems to have diminishing returns at some point. You get somewhere it’s like, nah this thing isn’t improving results anymore its just making the code like undeservedly long and annoying to pan through.
Layer type- This one. Dense layers seemed to work on my current dataset initially, but they just like immediately became frustratingly inept once the complexity of the data became significant.
It actually took me a while to attribute the absence of improvement to my choice of layer. I started doing all sorts of stuff. Increasing layer width, adding Dropout – some cool AI guy advised that–> Use the largest networks you can and then insert Dropout to countervail the deleterious memorisation tendencies consequent of sheer size. At least that was how I interpreted his suggestion.
I tried that. Meh. Network didn’t learn jack.
After like days of frustration and repeated modifications of the approach- each accompanied with an unfulfilled anticipation of progress, I thought to myself- You know, maybe Dense layers are just not the best choice.
So I switched to Convolutional layers. And I was like hm, you know this layer choice actually makes sense. But come on, I’m only here because my previous approach did not work. If Dense layers had worked bro I wouldn’t be here right now. And Dense layers made sense to me too when I decided to try them. When they did not eventually work and I was fairly certain I had tried like every possible sub-modification I could come up with, I then found myself constructing a narrative to explain the ineffectuality of Dense layers for my specific dataset.
Hm.
Is that what learning from experience is? Coming up with explanations for why something failed?
Brooo I spent days trying to figure out how to connect a number of Conv layers in a way that would be able to take in the intended data.
And then feeling considerably accomplished, I ran it.
It did not work.
Like oh man, Dense layers were better. At least they worked for extra-simple data. These ones aren’t doing jack. I wonder what the point of all of my “Hm you know Conv layers actually make a lot of sense for this dataset” is now. Like what’s the point of all of my logical justifications if this one still does not work.
Ughhhh.
I realised I needed to flip the way I was thinking about the convolution. The windows pan across the columns in my diagram, and not the rows.
Ohhh.
Come to think of it, that actually makes sense!
Hmmm!
Ugh again.
It’s like I just keep finding explanations for why initially encouraging ideas eventually did not work. But again, I feel like that’s like what learning from failure is.
Oh wow, there’s improvement. There’s freaking improvement. It works! Of course it does not work perfectly, but it shows potential. The window of effectuality in terms of dataset complexity is like waaayy wider than it was for Dense layers. This one actually makes sense.
Then came the hyperparameter tuning. Ugh. I feel hell could really just be the devil giving everyone arbitrary datasets and arbitrary network architectures and like task-mastering them to find optimal hyper-parameter configurations for the random problems.
Like seriously. There is very little logic to hyper-parameter tuning a lot of the time. Expanding layer width does not help with this dataset. I learnt that earlier so I don’t even bother with that.
Optimizer choice. This one is freaking random. How the hell am I supposed to know if Adam will work better than Rmsprop? Okay Rmsprop is supposed to be the best choice for sparse datasets but like, there were times the opposite seemed to be the case.
This life is messed up bro. Nobody really knows anything.
At this point I’m really just trying each of the different available optimisers and hoping the optimiser which’ll give me intended results already exists. If it doesn’t, I’m in trouble.
This life does not make sense. Engineering at this point is really just trial and error.
And development time is like soo long, because you’ll need to train for an appreciable number of epochs to even have an idea if the current configuration works better than the last one.
I’m working with a Linux cloud server with 224GB of RAM and 4 NVIDIA Tesla M60 GPUs. With a significantly complex model and appreciable dataset size you’re talking like 4-5 hours to know the effects your most recent model changes have on performance. And that’s like barest minimum time to just like have an idea if what you’re doing makes sense.
Now I feel like the people they talk about in Computer Science textbooks who had to wait for hours to get the results of executing their punchcard programs on the ancient workstations that preceded modern computers. Like hey here’s this program that’s supposed to add and multiply a few numbers would you mind running it for me while I go paint the house? Oh yeah cool thanks, I’ll be back like next tomorrow. Running programs was like dry-cleaning I think.
Hm, maybe sometime in the future I’ll look back at what I’m going through now and be like mehn those were the olden days of deep learning when you had to wait for hours to get computation results. Hah. We’ll see how things go.
And come to think of it, training deep learning models is really just adding and multiplying numbers. For the most part I believe.
So there are even more powerful Linux servers available, but I’m waiting on Microsoft to whitelist my access to those machines. Although come to think of it, the RAMs of those machines are double this one, but this specific model training is more GPU than CPU intensive. Increases in CPU capabilites do not really have an effect at this point since GPU number and capabilities are the same across both servers.
I realised this when I decided to multi-thread across the 224GB RAM CPU I’m working with. I increased the number of workers from 1 to like 20. There was absolutely no effect on training time. Like, I could see the computation being split across multiple threads, that just did not have any effect on training time.
I think if you’re training on GPU, CPU is only used for like loss computation per epoch or something like that. Or some online data preprocessing- like this Image Data Augmentation thing I saw somewhere.
Ugggh.
Oh wait. GPU models are not the same across both servers. The new ones are going to have the NVIDIA Tesla V100 GPUs. According to the info, that’s like 3 times more computational power.
Ohhh. Ohhhhhhhh.
Ahhhhh.
Okay. Okay. I’ll be anticipating the whitelisting then, haha.
Haha!
Hmmmm.
Regularization seems to be it. I’m getting training losses of like 11 point something. Data reconstruction is encouraging. And the model hasn’t even begun to over-fit yet. Please please please, a training loss of 11 is encouraging given my experience with this task. I started from a training loss of like 45 at overfitting. Started from the top and now I’m here.
To appreciate where you where, you need an awareness of where you’re coming from.
Mayowa Osibodu's Blog. 