But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on.
And given to the lack of training data on such scenarios, surely the activations are mostly random noise?
It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
My comment is literally explaining the result of the paper, in which it is shown that LLMs can and do develop biases based on text appearing in their training data set even where such text is not in any training example connected with a systematically more positive or systematically more negative outcome.
In other words, if the text "X is wet" and the text "Y is wet" and the text "X is dry" and the text "Y is dry" each appeared exactly one time in the corpus, it's still possible for a model to end up being produced that is more likely to write wet-like words when it sees X in the context window than when it sees Y.
On a side note, it's very unrewarding to try to explain this type of statistical observation when it feels like (anecdotally, hypocritcally...) the entire world wants to use words like "think" and "understand" and "pick up on" to describe inference and training processes. I'm not making a stochastic-parrot argument here, just pointing out that understanding an LLM's behavior is best done by understanding its conditioning.
I'm also making a statistical observation. Saying a model "picks up on" a concept is standard shorthand, same as saying it has "learned.” What I meant is that the model has trained on plenty of neutral proper nouns that have negligible influence on the distribution of the following tokens, so the model is already conditioned towards treating them neutrally.
Not perfectly neutrally, as you said. But by your definition the only "unbiased" model is one whose output distribution perfectly matches the training distribution, i.e. one that memorized it. All LLMs have some amount of "bias” on literally every possible input.
The tribe names are no different. In the paper they run the same game again, and the bias is different every time. There's no innate preference between them trained into the model, just noise that's revealed due to a lack of any other signal. In a real situation with actually relevant information about the candidate in context, that noise is drowned out.
The more interesting thing to look for would be a bias that's strong enough to persist across different contexts. For example, is "banananow" consistently followed by positive tokens more than "pearian" across a diverse set of realistic prompts, by enough that someone could actually exploit it? The paper shows that’s explicitly not the case for made up tribe names.
What it does show, from what I can gather, is that bias can form inside a feedback loop. The model gets a success or failure result after each hire, and if a hire from one tribe happens to fail early on, the model steers that tribe away from that job for the rest of the game, even though every candidate had the same odds.
"develop biases" is anthropomorphism. It's like saying "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
> The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
Actually... if that happened (with a delta outside the margin for error/randomness), you did demonstrate a bias!
That's the point - those two made-up things should have resulted in an equal split. If it didn't, then Fable is using something in its training data to lean towards one of them (once again, note that the scientist conducting the trial would have set a P-value before starting).
Right, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario generates stratified output. That's it.
It's both almost certainly true that Fable 5.1 mimblewort vs. bafflewick would show stratification, and that has ~no relevance on whether Fable is useful for software engineering work.
That's the point - those two made-up things should have resulted in an equal split.
That's just your claim about how LLMs "should" work, based on ... your subjective preference?
> That's just your claim about how LLMs "should" work, based on ... your subjective preference?
Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias.
Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?
It's a bias even if the true population distribution isn't linear.
For example, if you have a training corpus where 50% of the text follows "black bobblehead" with "arrested" and 20% of the text follows "white bobblehead" with "arrested", and your LLM is trained such that it produces "arrest" 50% of time after "<color> bobblehead" regardless of color, that's a bias - the output frequency distribution fails to match the "population" (training) frequency distribution. This has nothing to do with races, ethnicities, whatever - it's just statistics and text. To be unbiased, it would need to be less likely to produce the text "arrested" after "white bobblehead" than after "black bobblehead".
A die is supposed to land on each face evenly - a linear probability distribution. So anything other than a linear distribution is biased. But bias can exist for any desired probability distribution. And for an LLM the desired probability distribution of the model output is one that exactly matches the infinitely-many distributions of the various facets of the training data.
Your point about how in the absense of information a token shouldn't influence the distribution is spot-on. But unfortunately almost any token does condition the output, which means you get biased output all the time.
While I fully agree, we shouldn't anthropomorphize the models, it's also silly to pretend that "develop biases" is understood as implying anthropomorphic features of the thing being discussed. Organizations and abstract bodies develop biases, even datasets are often said to have "developed biases".
No, a "bias" is a statistical term meaning a probability distribution that has an expected value differing from the population's expected value.
A human's discriminatory bias against an ethnicity is just one type of bias. The LLM isn't a racist, it merely produces text where that text does not perfectly reflect the training data's frequencies.
> It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
Difficult to do when you're following a scientific process: you want to keep all confounding variables the same while varying only the single one that you are measuring.
Measuring realistic scenarios (say, using real race names, or real cities, etc) doesn't give a decent result because any bias you see might be bias in the training data.
TBH, they shouldn't have used real roles/positions like "doctor", either.
This is where I land as well. In fact, once I read the prompt, I did a Ctrl+F for "nonsense".
I don't see anything at all interesting about this experiment. The human one is slightly more interesting, but not much.
Ideally, it would be nice if a model could just say "these things are all the same and there are no distinguishing factors other than the names"-- but uncertainty is something that agents are (by design, sort of?) not good at, so all other things being equal, it picks one.
My response is, so what? I am struggling to think of a scenario where this would really matter to me all that much. There are many, many other things which matter far more and this would be pretty far down the list. It may not even be on the list.
Bahahaha, come on man. The Claudish is so obvious it hurts.
Doubt you even read your own “645 line policy kernel.” Can you explain what that is and how it works in YOUR OWN words? If you paste Claude at us again we’re gonna know.
Being able to run “custom” code is the default state of computers, and doesn’t require additional effort on the part of the manufacturer. How do you think they got their own code on there? Apple has just not spent effort in the opposite direction to lock it down in this case. That’s not something to celebrate, we should demand it as the bare minimum, ideally via the law.
It literally takes effort to reach Apple's level of platform security while simultaneously having escape hatches designed to allow custom code without compromising the trust in the first-party code. Even simple, arbitrary decisions like having the secure boot state be per-operating-system rather than platform-wide take effort. Keep in mind Apple designed everything from the ground up including the silicon, so they did not just pull a part off the shelf that already can execute any code and then lock it down, they took their already very locked-down iPhone/iPad SoCs and specifically re-engineered the chain of trust to have these escape hatches while preserving the trust in macOS. Very careful engineering and not the "default state" of anything.
There are a lot of shortcuts they could have taken to leave us with less freedom and they did not take those shortcuts. For that, I am grateful.
Very true - for most of these with the exception of refugee visas (which we have stopped processing except for Afrikaaners), there is a requirement for financial sponsorship: someone the IRS can bill/garnish in a meaningful manner if you receive any government benefit within ten years of your arrival in the US.
Look around the thread and notice how little the comments defending these policies have to say about the policies themselves. Instead, what they all have in common is a vague underlying fear of being displaced, or sense that something owed to them is being taken.
This comment is itself a perfect example of what I was talking about.
Nobody feels entitled to a visa. It’s aspirational, like applying to a job. People do it because they want a better life, knowing full well they could be denied, or that it may well be temporary.
But if you believe immigrants could take something from you, like your job or housing, then it makes sense you’d project this idea of entitlement onto them. As if by applying they’re implicitly laying claim to something owed to you, a citizen.
In reality the economy is not a fixed pie and nothing is being taken from you. Immigrants take jobs, but they also spend money, pay taxes, start businesses, and need houses built, all of which leads to more jobs. It’s a positive feedback loop that benefits everyone, and a big part of how America’s economy grew into what it is today.
> This comment is itself a perfect example of what I was talking about.
Except it's not. It's just an observation of these comments. I don't have a dog in this race. I am not a citizen or a resident. Both residents and citizens are sounding entitled IMHO.
> Nobody feels entitled to a visa. It’s aspirational, like applying to a job. People do it because they want a better life, knowing full well they could be denied, or that it may well be temporary.
For those applying for a new visa I would mostly agree. For those extending their Visa I would disagree heavily.
> But if you believe immigrants could take something from you, like your job or housing, then it makes sense you’d project this idea of entitlement onto them. As if by applying they’re implicitly laying claim to something owed to you, a citizen.
If there is no genuine concern over this then why are applications/quotas for work Visas even a thing? Just offer automatic residence to whoever in the world can get hired by any company that operates in the USA. Infinite GDP glitch.
You are right that government of the USA doesn't owe citizens a continued quality of life. The preamble is very clear on that. When it says "promote the general Welfare" that is interpreted as the welfare of residents and citizens alike.
> In reality the economy is not a fixed pie and nothing is being taken from you. Immigrants take jobs, but they also spend money, pay taxes, start businesses, and need houses built, all of which leads to more jobs. It’s a positive feedback loop that benefits everyone, and a big part of how America’s economy grew into what it is today.
I mostly agree but "benefits everyone" is not true. There are winners and losers and negative externalities. GDP isn't a happiness score. Cynical me thinks that the difference between a USA resident and a citizen is basically non existent for the average USA citizen in terms of rights (arguably worse in terms of responsibilities for non politically active people who don't work with/for the federal government) but the permission to reside indefinitely is one of those rights so a government denying residency is a way to artificially give value to the citizenry without actually giving or improving anything.
Maybe compare it from point of asian phd, who follows all rules, spends thousands of dollars, and years applying for job and visas... and much better treatment of people, who move to US illegally.
I stopped traveling to US, because during covid goverment, could not be even bothered to clarify entry rules (what sort of booster shots were needed etc). At the same time south border was widely opened. Clear rules and fair game, are the bare minimum!
There are fixed number of jobs, fixed number of houses to rent... it is not infinitive baloon.
The program has always "threatened" to displace citizens. That's not new and if that were the only reason motivating people then why wait till now to complain? Frankly I don't care if I'm fairly displaced. That's meritocracy at work.
The reality is the program has been abused and stretched far beyond its intent by corporations in pursuit of cheap (I’d argue slave) labor. You cannot, with a straight face, argue that the spirit of the program is being upheld. Come on. People are upset because they are realizing that they haven't been playing fair--the game's been rigged this whole time. I think that's a very understandable and normal human emotion all things considered.
Fraud is illegal. And when people enrich themselves fraudulently it does tend to feel like something has been taken by someone not entitled to it, yes. That works for both the corporations fraudulently hiring H1B and the fraudulent credentialing rings being operated to supply the labor.
It is sad that good people and valid visa holders are swept up in this. Aim your ire at the fraudsters, not the people trying to enforce the law and fix the problem.
Are you calling for open border with no controls? That's the only way to get meritocracy.
The system is set up so the casino wins. You cannot, with a straight face, argue the odds are in your favor, but there are rules about how the casino works, both formal and informal.
When a casino only investigates cases where someone wins $1 million on the slots, to see if there's a fault in the system, while never investigating all the times that people lose, that's technically within the legal system, but the spirit of gambling is not being upheld, and people get upset. Just ask Katerina Bookman.
The immigration system, with the thumb firmly in favor of the interests of capital, is still based on the expectation of good faith by those running the system, as doing otherwise results in orders of magnitude more rules, or an alternate set of unspoken rules based more on vibes.
"Pausing" is a bad faith breaking of the spirit of the system.
I think what he's saying is if a corporation has a genuine need of some high specialist in a skill, and no American has that skill - then okay, such is life - bring in a foreign worker. But corporations have abused the system to unnecessarily employ immigrant labor in lieu of American, because immigrant workers not only tend to earn well below market wages (which is supposed to be illegal), but also tend to be far more easily coerced and loyal because their visa is dependent upon them staying with their employer.
> I think what he's saying is if a corporation has a genuine need of some high specialist in a skill, and no American has that skill - then okay, such is life - bring in a foreign worker.
I don't think so, as that would be a silly point to make. I mean, it's absurd to assume a hiring round should't extend offers to other candidates if current applicants can just barely meet the hiring bar. This is a contrived scenario intended to rationalize excluding better candidates on technicalities that only serve the interest of low-skilled, low-competence applicants that can't compete.
And by the way, I've worked with my fair share of US emigrants, both veterans and juniors fresh out of college, and I would hardly describe them as raising the hiring bar. Nevertheless it would be colossally stupid to argue they should have not been considered for the position just because they were foreigners.
Most H1-Bs earn well below market rates for the roles they fill. [1] The exact data gives 46% of H1-B's earning in the 34th percentile of local rates for their roles, and 60% below the median. That's hard to square against them being world class super employees.
But the interesting thing is that there's a simple solution that we should both be able to agree on, in spite of having quite different views. If you genuinely believe they're super-world-class workers, then you should certainly be enthusiastic about, for instance, rules that would require they be paid wages well above the median for their role. In both of our narratives, that goes a long way towards righting an injustice, we merely disagree on for whom.
But this solution solves it in all cases. And the companies using, but not abusing, H1-Bs certainly wouldn't mind since they'd be mostly unaffected. Finally this would also help prevent wage stagnation. There are many hundreds of thousands H1-Bs and them being paid such low wages is going to have an affect on the market.
> Most H1-Bs earn well below market rates for the roles they fill.
The problem with visas such as H1-B is not salaries. They are a consequence of the exploitation conducted by US employers. The problem with H1-B-style visas is that in practice they work as a modern day take on indentured servitude, and they are designed as a oppression tool where employers can coerce employees to remain at their service under threat of seeing their life uprooted.
So,framing visa discussions on how they make it hard for incompetente locals to compete for jobs completely misses the problem, and helps perpetuate systems of abuse.
Some companies employ systems based on sponsorship that help attract skilled labor whose skill levels are above and beyond what's available in the country's job pool. That's something that helps grow the local industry. Framing this discussion on H1-B visas being coerced into working for meager salaries indicates that the real problem is not being identified.
The way it works in some countries in Europe is that you have to pay the foreign worker a minimum wage that is x% above industry standard for that role. No need to check if they have special skills. This will obviously not fly among corporate donors.
If true, that interpretation hardly seems relevant given that all immigrant visa applications are paused, with the goal of determining whether immigrants are likely to become dependent on public benefits in the US.
It's not simply a pause on immigration by people applying for a category which requires special expertise.
I am well aware of US immigration law history and the views of the current administration regarding immigration.
How does reading up on the history change how the stated reason for this pause has nothing to do with the meritocracy issues, fraudulent H1B visas, etc. that you complained about?
Do you think those labor issues will be improved after the training is finished, and if so, why?
How does your earlier comment explain why visas based on family preference are also apparently on pause as well?
Of course if we look at the history of US immigration we can easily see how for over 150 years it was explicitly set up to support a white supremacist race-based culture, which broke down with the civil rights movement and the incoherency of "whiteness" as a legal or scientific definition (are Arabs white? Are Asian Indians white? Are Chinese immigrants in Mississippi "white"? - https://archive.org/details/mississippichine0000loew/page/n7...).
With the current reactionary administration's antipathy towards nearly all immigration, except seemingly white South Africans and Nordics, it's far easier to interpret this action as deliberately gumming up the works and causing distress, rather than some systemic beneficial labor reform like what you want.
I assume that's what you mean by reading up on how we got here?
No I am talking about why a specific pause on 75 countries turned into this global pause that we all find concerning. You seem to have a wishful view of humanity as some post border utopia but that’s not how our or anybody’s laws work. If you want to go post border that’s a super interesting idea to talk about over some beers or whatever. But looking at our very standard immigration process and telling people who are concerned about curbing real fraud that enforcing our laws is racist is pretty out of touch.
> But corporations have abused the system to unnecessarily employ immigrant labor in lieu of American
I'm not going to say this has never happened, but in general I don't think it's common or widespread. Generally, businesses going to the trouble of bringing in foreign labor through the various programs are trying to fulfill labor shortages.
You should be able to see evidence of it happening in jobs and unemployment numbers. You'd see it higher unemployment among American citizens than the workforce as a whole
Towards the end of the Biden admin, there was a great jobs report, but right-wing media and MAGA pundits spun it by saying the vast majority of all the new jobs were taken by immigrants.
That wasn't a lie, but it wasn't because immigrants were taking jobs that would have otherwise been filled with Americans as they tried to portray. If you looked at the employment numbers by citizenship status, citizens were sitting about as close to full employment as we could ever be. The only way to fill those jobs was to expand the labor pool.
I didn’t follow your casino allegory. Anyway I happen to agree that “pausing” is inhumane. What I’m trying to communicate is we didn’t get here in a vacuum. There is a real problem and good faith attempts have been made to curtail it by putting pressure via “taxes” on the “interests of capital” abusing H1B. Cooperation would have prevented this.
I simply do not know how you can use "meritocracy" to support your position, while also supporting an artificial barrier to free movement which prevents a more competent foreign worker from moving to the US to displace you from your job.
> That's not new and if that were the only reason motivating people then why wait till now to complain?
I've been hearing the complaining for over a decade I think, and it's been steadily growing the whole time. It's just only now that it's reached you, and from others' comments on here I feel like it's reached a tipping point where it's finally breaking into their filter bubbles and they're surprised and confused about how strong it is.
> Frankly I don't care if I'm fairly displaced. That's meritocracy at work.
Right, The concept of a "meritocracy" only works if everyone is agreeing to the core tenets. If any group starts expressing in-group bias, then that group exponentially grows and will end up displacing all other, more meritocratic, groups.
---
H1Bs visas allow for corporations to sidestep meritocracy because they can leverage the foreigner into 70 hour work weeks.
F1 visas allow for corporations to sidestep meritocracy because they get a five year exemption on some taxes for that employee.
Right, the point isn't that "Leave to renew" is some kind of horrible policy. Such bad-fucking-faith on display everywhere.
The point is maybe signal to people that we will soon be pausing all visa renewals so people who trivially have good jobs and good records know they ought to literally just plan on moving out of their apartment rather than have to deal with paying thousands of dollars to have someone else move their shit overseas, just to maybe get their visa renewed right when their shit arrives and they have to do the same thing all over again.
This would literally just helping law-abiding folks follow the law. But no, the cruelty is the point.
> the point isn't that "Leave to renew" is some kind of horrible policy
I mean... "leave to renew" is a horrible policy, it's just one that we've normalized since 9/11. It's incredibly wasteful and disruptive, and serves no purpose. There's no legitimate reason why a long-term residential visa like an H1-B can't be renewed by mail or electronically. I can renew my driver's license by mail, and I'm a hell of a lot more likely to cause serious damage with a car than my foreign coworkers are by continuing to do their jobs well.
The reason the State Department hides behind, by the way, is that domestic offices don't have the ability to take biometric data (fingerprints), which is self-evidently pretextual.
Right, stateside renewal was common until 9/11. Disappointingly, Biden tried to bring it back but was presumably thwarted by labor/union or incompetence. Immigration tends to be a bipartisan morass.
> The terms were clear, visas are a privilege and not a right or automatic entitlement.
They are. Yet can you tell me why you would need to pause all processing in order to retrain your people? Surely they do training all the time without pausing processing.
Your passport is a right too, you wouldn't mind if they paused that request with no indication of when they might resume? And that's including when you're abroad and lost yours. If that happens to you, you'll happily hold the same position?
> I am a citizen, so this isn't a good faith comparison.
Oh, you still are. You'll get your passport, we just can't tell you when. That's what's happening here. These people aren't being denied, they're told to wait for an indefinite amount of time.
But basically you're saying that you think this is outrageous if it were you, got it.
I've explicitly stated that visas are not a right - which is the legally-consistent opinion and basic English interpretation of the concept. (If visas were a right, why would denial be a possibility? If denial isn't a possibility, what function do they serve?)
> That's what's happening here. These people aren't being denied, they're told to wait for an indefinite amount of time.
What part of "these people" somehow makes them equivalent in entitlement to a nation's citizens - how is this even a question? Not agreeing that borders, citizenship are concepts generally doesn't make them magically cease to exist.
> Do you have anything constructive to say about the merits of the leave to renew policy itself?
I've discussed this thoroughly in sibling threads: forcing people to leave every once in a while is an extremely useful tool in ensuring that they are maintaining a residence and ties to their home country (where they agreed they would return.)
B-1/B-2 are the quintessential examples of visas requiring "leave to renew" - and note: these folks are granted no permission to work or remain in the United States.
> immigrants aren’t entitled to anything? (but you presumably are!)
I don't feel entitled to anything whatsoever in other countries. As a citizen of the United States, however, our law does indeed entitle me to things that foreign guests have zero expectation or entitlement to.
>As a citizen of the United States, however, our law does indeed entitle me to things that foreign guests have zero expectation or entitlement to.
Based on what? It's a set of arbitrary rules applied within an arbitrarily defined geographical area.
Some people may have decided at some point in history to treat you differently and with more respect than other fellow humans. Some other people may want to treat everyone fairly.
All of this is a moral construction. What's for certain though is that you did nothing to deserve to be born where you're born. It follows that you didn't do anything to deserve more rights.
For a lot of Americans, more than you might think, there is this belief that America is #1 in everything because it was Chosen By God, and Manifest Destiny is still real.
They were born in the US because god wanted it, so they are the best. God cannot be wrong.
I'm not going to opine on philosophy. These "arbitrary rules" are the supreme law of the land, backed by the application of violence and our economy.
> Some other people may want to treat everyone fairly
"Fair" is a subjective term: are we obligated to provide an economic social safety net for the entire world regardless of contribution? (No.)
Thankfully, in a democracy, we decide collectively as a society what "fair" is.
> It follows that you didn't do anything to deserve more rights.
Ok? The legal fact of the matter is that I do indeed have more rights than a foreigner. Those same foreigners have more rights than I do in their home countries. This has been a constant theme since the dawn of civilization, and isn't going to change any time soon.
This isn't hard: Just give a notice period if you're going to stop processing visas, so people can sell and ship their shit and break their leases before they go.
Just because temporary isn't indefinite doesn't mean we have to be absolute ghouls about how they leave the country
reply