Serving the Quantitative Finance Community

• 1
• 2

cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

### Python: lambda functions.

I'm trying to find an easy way of calling a UDF in a lambda function in dataframe:
Here's the dataframe:
df = pd.DataFrame({'A': ['foo', 'bar', 'baz', 'foo'],
'B': ['qux', 'quux', 'quuz', 'xyz']})
df
Now the objective is to search for all rows in both columns that meet a specific criteria (user defined) and create two new columns based on that. The first new col will have a boolean value, and the second column will have a specific text value.
First: here's what works: I write the functions directly and they work. Col 2 and 3 both have the correct boolean values; based on the search conditions, only the first row returns True, all the other rows are False.
df['col2'] = (df['A'] == 'foo') & (df['B'] == 'qux' )

df = df.assign(col3 = lambda x: (df['A'] == 'foo') & (df['B'] == 'qux' ))


Of course, in the real world dataset, conditions become more convoluted. It's easier to have a separate UDF listing all the combos and then call it separately. To that end, I wrote the UDF:
def test(df):
if (  (df['A'] == 'foo') & (df['B'] == 'qux' ) ).any():
x = True
else:
x = False
return x


and then called it two different ways.

df = df.assign(col4 = lambda x: test(df))
# df['col5'] = df.apply(test, axis=1)
df['col6'] = df.apply(lambda x: test(df), axis=1)

As you can see from the output, the function spits out True for all rows - clearly wrong. I suspected that maybe the "any()" operator is creating some problems.....but if I remove that...I get the "...truth value of a series is ambiguous..." error.

Would appreciate any suggestions!!

katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

### Re: Python: lambda functions.

Why not simply
def test(df):
x =  (df['A'] == 'foo') & (df['B'] == 'qux' )
return x 
??

cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

### Re: Python: lambda functions.

Thanks!
Took me a while to figure this out - but  it looks like there's a very specific way to call the lambda function and this is where I was going all wrong.
Using the code you provided (and modified for a more general case), I ran the following:

def test5(df):
if  (df['A'] == 'foo') & (df['B'] == 'qux' ):
x = 'peak'
elif (df['A'] == 'bar') & (df['B'] == 'quux' ):
x = 'off'
else:
x = 'false'
return x

# this works - have to use apply not assign.
df['col10'] = df.apply(lambda x: test5(x), axis=1)

# this fails "truth value of series...." error; assign doesn't work.
df = df.assign(col11 = lambda x: test5(x))
So it took me a while to figure out that I need to use apply and not assign. I found that the assign method gave no errors when I modified the code as follows:

def test6(df):
if ( (df['A'] == 'foo').any() & (df['B'] == 'qux' ).any() ):
x = 'peak'
elif ( (df['A'] == 'bar').any() & (df['B'] == 'quux' ).any() ):
x = 'off'
else:
x = 'false'
return x

# this "works" - code doesn't fail, but has the wrong answer

df = df.assign(col12 = lambda x: test6(x))


But of course now, it doesn't give me the correct result - all rows are marked with the same label (and the use of "any" is the issue....while it prevents the "...truth value of series is ambiguous error...", it produces the wrong output).
So after a lot of googling, I figured your code combined with the correct way to call the lambda function worked. But it's been a terribly confusing and annoying journey so far....I'm glad it's over! Thanks for the help again.

katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

### Re: Python: lambda functions.

The only difference between the two methods is that apply will modify the column on which you call it, and assign will add a new column.
I think you might need to check .any() does what you want. It checks if *any* element in the whole column is equal to the string. If you want to check each row separately, skip any() as I suggested above. But I only skimmed your posts, so I may not understand what you're doing

cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

### Re: Python: lambda functions.

Thanks sir - any AND the way I was calling Lambda was creating the issue. But your post was super helpful - I had been staring at my own scribbles for so long that I just couldn't spot the issues. Again - appreciate the time!

katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

### Re: Python: lambda functions.

No problem.
You might want to try using Copilot or even ChatGPT to get help with such issues, BTW.

cdsharm75
Topic Author
Posts: 26
Joined: November 5th, 2020, 6:40 pm
Contact:

### Re: Python: lambda functions.

ah....totally forgot about that......will keep in mind for next time. Thanks!

jasonbell
Posts: 8
Joined: May 6th, 2022, 4:16 pm
Contact:

### Re: Python: lambda functions.

Python 3.10 now has match/case functionality. A lot neater and easier to maintain that a raft of if/then/else statements.

Personally I'd be careful with that & within the if statement, using "and" is safer.
Founder of Synthetica Data - https://www.syntheticadata.com
Author of Machine Learning: Hands on for Developers and Technical Professionals (Wiley).

tagoma
Posts: 2901
Joined: February 21st, 2010, 12:58 pm

### Re: Python: lambda functions.

Python 3.10 now has match/case functionality. A lot neater and easier to maintain that a raft of if/then/else statements.

Personally I'd be careful with that & within the if statement, using "and" is safer.

Hi JB. Wha is AND safer than & sign, please?

katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

### Re: Python: lambda functions.

Bitwise operators don’t short-circuit

jasonbell
Posts: 8
Joined: May 6th, 2022, 4:16 pm
Contact:

### Re: Python: lambda functions.

Python 3.10 now has match/case functionality. A lot neater and easier to maintain that a raft of if/then/else statements.

Personally I'd be careful with that & within the if statement, using "and" is safer.
Python 3.11.5 (main, Aug 24 2023, 15:08:51) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
>>> a = 200
>>> b = 33
>>> c = 500
>>> if a > b and c > a:
...   print("Both conditions are True")
... else:
...   print("This is false!")
...
Both conditions are True

>>> a = 200
>>> b = 33
>>> c = 500
>>> if a > b & c > a:
...   print("Both conditions are True")
... else:
...   print("This is false!")
...
This is false!
>>>

^^^^^
Hi JB. Wha is AND safer than & sign, please?
Hi Tagoma, straight from the CLI. And we're expected to trust sklearn
Founder of Synthetica Data - https://www.syntheticadata.com
Author of Machine Learning: Hands on for Developers and Technical Professionals (Wiley).

tagoma
Posts: 2901
Joined: February 21st, 2010, 12:58 pm

### Re: Python: lambda functions.

bitwise vs boolean. thanks guys.
no animal was hurt.

katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

### Re: Python: lambda functions.

Operator precedence above is one basic rule to remember, and here's an example of tragic consequences of using bitwise operators in short-circuiting that I mentioned before:
def cook1(recipe):
if "salt" in recipe:
return False
else:
return True

def cook2(recipe):
if "salt" in recipe:
return False
else:
return True

recipe = ["salt", "flour", "water"]

print("Bitwise AND:")
if cook1(recipe) & cook2(recipe):
print("Voila!")

print("\nLogical AND:")
if cook1(recipe) and cook2(recipe):
print("Voila!")
Bitwise AND:
Logical AND:

Bitwise cooks added twice as much salt! :(

How low have we fallen. We are discussing the basics of Python... In the past we argued about accuracy and precision of automatic differentiation vs complex step method, alternatives to modules for encapsulation in C++, non-determinism of monads in the context of Ancient Greek atomism, ... All those moments will be lost in time, like bits in NADN gate. Time to sleep.

Cuchulainn
Posts: 19117
Joined: July 16th, 2004, 7:38 am
Location: 89 19 79 15

### Re: Python: lambda functions.

katastrofa
Posts: 7185
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

### Re: Python: lambda functions.

A bit of an overkill in general, imho, since short circuiting in coding is used for efficiency.